RBA Cash Rate: 4.35% · 1AUD = 0.67 USD · Inflation: 4.1%  
Leading Digital Marketing Experts</strong | 1300 235 433 | Aggregation Enquires Welcome | Book Appointment
Example Interest Rates: Home Loan Variable: 5.69% (5.89%*) • Home Loan Fixed: 5.39% (5.84%*) • Fixed: 5.39% (5.84%*) • Variable: 5.69% (5.89%*) • Investment IO: 5.69% (6.19%*) • Investment PI: 5.55% (6.02%*)

How to Use the Abuse and Malware API

How to Use the Abuse and Malware API

Some time back we had a website with an 'unauthorised' plugin introduce sophisticated Malware into a website that resulted in damage that made its way into other system files. As a result of that breach we introduced additional measures to mitigate the impact of potential intrusions - a necessity given the sensitive nature of any financial asset.

Every server can be penetrated in some way, and it's only a matter of time before one finds themselves victims of a malicious actor. We create daily website backups so the restore process is always easy, but the process of tightening server functionality is one that is always ongoing. One of the more recent security measures we introduced makes use of our Abuse API that will simply ban any incoming request that is in any way associated with known, reported, or suspected illegal activity or abuse.

The Abuse (Malware) API is updated every hour with hundreds of thousands of records sourced from multiple vendors. If an incoming request matches a suspected source, that request is blocked and a '403 Forbidden' message is returned. This article will details how the Abuse (Malware) API is used.

No Solution is Perfect: Malware practitioners will often scan websites for vulnerabilities, crawl websites, or brute force login screens or other files that carry a weakness. The business of Malware is a sophisticated one so our methods (when used in isolation) isn't perfect, but it's a reasonable measure for blocking known malicious sources, and a highly effective tool when used in company with other tools.

Website Backups: We make client website backups every day, and Yabber provides a facility to create your own backup when required. The offsite backups are restored by us when necessary. This style of service will normally carry a significant fee but is provided to our clients at no additional cost. Well over 40% of all websites on the Internet are created with WordPress but like any open-sourced application, this make the software vulnerable to clever hackers. We believe that if we're going to use WordPress we need to provide a reliable and industry-leading framework for backups (in addition to providing a service where those backups are unlikely to be required).

  Abuse (Malware) API

In addition to returning details on IP addresses and Network IPs that are associated with malicious activity, the Abuse and Malware API will also return details for identified web crawlers (many of which are used by the malware crowd). The API is a standard client-only RESTful API that is now integrated with about every application we create.

  API Endpoint

The API supports a number of operations, but the standard GET request is most commonly used and is accessed via the following endpoint.

api.beliefmedia.com/abuse/abuse.json?ip={ip_address}&apikey={apikey}

Access to the API is made available via your primary API Key. No limitations are applied to use, but if you expect more than 30 requests per second we'd appreciate an understanding of your application. If accessing the API from a service that is not hosted by us you will also be required to supply a k parameter which is a mutating string valid for no more than a few seconds.

This article will introduce a few of the expected API responses.

  Examples

Let's consider the IP address of 45.80.158.62. The request returns JSON that unfolds into the following array (truncated for readability).

1
Array
2
(
3
    [is_malware] => 1
4
    [is_bot] => 0
5
    [status] => 200
6
    [data] => Array
7
        (
8
            [ip] => 45.80.158.62
9
            [network] => 45.80.158
10
            [bot] =>
11
            [network_count] => 53
12
            [network_ip] => Array
13
                (
14
                    [45.80.158.69] => Array
15
                        (
16
                            [ip] => 45.80.158.69
17
                            [network] => 45.80.158
18
                            [host] => 69
19
                        )
20
 
21
                    [45.80.158.81] => Array
22
                        (
23
                            [ip] => 45.80.158.81
24
                            [network] => 45.80.158
25
                            [host] => 81
26
                        )
27
 
28
                    [45.80.158.74] => Array
29
                        (
30
                            [ip] => 45.80.158.74
31
                            [network] => 45.80.158
32
                            [host] => 74
33
                        )
34
 
35
                    [SNIP]
36
 
37
                        )
38
 
39
                    [45.80.158.128] => Array
40
                        (
41
                            [ip] => 45.80.158.128
42
                            [network] => 45.80.158
43
                            [host] => 128
44
                        )
45
 
46
                )
47
 
48
            [updated_at] => 1714870292
49
            [urlhaus] => Array
50
                (
51
                )
52
 
53
            [urlhaus_network] => Array
54
                (
55
                    [2787769] => Array
56
                        (
57
                            [2b088cf7a3f3651ba0b6cd67fc12d625] => Array
58
                                (
59
                                    [is_ip] => 1
60
                                    [created_at] => 1710947772
61
                                    [created_at_dt] => 2024-03-21 02:16:12
62
                                    [url] => http://45.80.158.168:222/x.jpg
63
                                    [ip] => 45.80.158.168
64
                                    [network] => 45.80.158
65
                                    [url_status] => offline
66
                                    [last_online] => 1711038499
67
                                    [last_online_dt] => 2024-03-22 03:28:19
68
                                    [threat] => malware_download
69
                                    [urlhaus_link] => https://urlhaus.abuse.ch/url/2787769/
70
                                    [reporter] => abus3reports
71
                                    [updated_at] => 1714886158
72
                                    [urlhaus_id] => 2787769
73
                                    [hash] => 2b088cf7a3f3651ba0b6cd67fc12d625
74
                                )
75
 
76
                        )
77
 
78
                    [2787766] => Array
79
                        (
80
                            [177d1a6fea037de676d9c9690e2fc2d8] => Array
81
                                (
82
                                    [is_ip] => 1
83
                                    [created_at] => 1710947769
84
                                    [created_at_dt] => 2024-03-21 02:16:09
85
                                    [url] => http://45.80.158.168:222/xt.txt
86
                                    [ip] => 45.80.158.168
87
                                    [network] => 45.80.158
88
                                    [url_status] => offline
89
                                    [last_online] => 1711037964
90
                                    [last_online_dt] => 2024-03-22 03:19:24
91
                                    [threat] => malware_download
92
                                    [urlhaus_link] => https://urlhaus.abuse.ch/url/2787766/
93
                                    [reporter] => abus3reports
94
                                    [updated_at] => 1714886158
95
                                    [urlhaus_id] => 2787766
96
                                    [hash] => 177d1a6fea037de676d9c9690e2fc2d8
97
                                )
98
 
99
                        )
100
 
101
                    [2787767] => Array
102
                        (
103
                            [ade9faed99bbca5078585dd821d6b60a] => Array
104
                                (
105
                                    [is_ip] => 1
106
                                    [created_at] => 1710947769
107
                                    [created_at_dt] => 2024-03-21 02:16:09
108
                                    [url] => http://45.80.158.168:222/jj.jpg
109
                                    [ip] => 45.80.158.168
110
                                    [network] => 45.80.158
111
                                    [url_status] => offline
112
                                    [last_online] => 1711038833
113
                                    [last_online_dt] => 2024-03-22 03:33:53
114
                                    [threat] => malware_download
115
                                    [urlhaus_link] => https://urlhaus.abuse.ch/url/2787767/
116
                                    [reporter] => abus3reports
117
                                    [updated_at] => 1714886158
118
                                    [urlhaus_id] => 2787767
119
                                    [hash] => ade9faed99bbca5078585dd821d6b60a
120
                                )
121
 
122
                        )
123
 
124
                    [2787768] => Array
125
                        (
126
                            [6d5589db60293c8ea4224af89b7b3060] => Array
127
                                (
128
                                    [is_ip] => 1
129
                                    [created_at] => 1710947769
130
                                    [created_at_dt] => 2024-03-21 02:16:09
131
                                    [url] => http://45.80.158.168:222/gs367.txt
132
                                    [ip] => 45.80.158.168
133
                                    [network] => 45.80.158
134
                                    [url_status] => offline
135
                                    [last_online] => 1711039781
136
                                    [last_online_dt] => 2024-03-22 03:49:41
137
                                    [threat] => malware_download
138
                                    [urlhaus_link] => https://urlhaus.abuse.ch/url/2787768/
139
                                    [reporter] => abus3reports
140
                                    [updated_at] => 1714886158
141
                                    [urlhaus_id] => 2787768
142
                                    [hash] => 6d5589db60293c8ea4224af89b7b3060
143
                                )
144
 
145
                        )
146
 
147
                )
148
 
149
        )
150
 
151
)

A valid request will always return a code of 200. The is_malware and is_bot keys will usually carry a value of 1 (true) or 0 (false), and it's these values that you'll generally reference for standard blocks. The network_count key will return the number of IP addresses reported within the network range (in this case, 53), and those IPs are listed in the network_ip array.

URLhaus is a website that provides a regularly updated list of Malware URLs, so we include those malicious URLs associated with the urlhaus_network key. The excellent URLhaus data is only made available for non-commercial purposes so it shouldn't be used or relied upon for anything other than your own personal applications. The URLhaus data is indexed on the URLhaus IP ID and each associated URL record is indexed on a hash which is an md5 hash of the IP address (without any port if one was provided). Our primary data is derived via the network_ip array... although, in reality, we rely almost exclusively on the single digit is_malware flag.

In the next example we'll have a look at an IP that resolves to the Chinese-owned ByteDance (parent company of TikTok). The number of requests from ByteDance is truly staggering - sometimes in the order of over 1000 requests every hour. Do we really need ByteDance to have access to our website? Probably not. Should we ban this bot? Probably.

1
Array
2
(
3
    [is_malware] => 0
4
    [is_bot] => 1
5
    [status] => 200
6
    [data] => Array
7
        (
8
            [ip] => 47.128.29.53
9
            [network] => 47.128.29
10
            [bot] => Array
11
                (
12
                    [bot_name] => Bytespider
13
                    [bot_category] => Bytespider
14
                    [bot_url] => https://bytedance.com/
15
                    [bot_producer_name] => ByteDance Ltd.
16
                    [bot_producer_url] => https://bytedance.com/
17
                    [bot_count] => 1539
18
                    [ip] => Array
19
                        (
20
                            [0] => 110.249.201.151
21
                            [1] => 111.225.149.90
22
                            [2] => 111.225.149.241
23
                            [3] => 110.249.202.139
24
 
25
                            [SNIP]
26
 
27
                            [1534] => 47.128.33.193
28
                            [1535] => 111.225.149.165
29
                            [1536] => 47.128.17.13
30
                            [1537] => 47.128.59.136
31
                            [1538] => 47.128.42.37
32
                        )
33
 
34
                )
35
 
36
            [network_count] => 1
37
            [network_ip] => Array
38
                (
39
                    [47.128.29.90] => Array
40
                        (
41
                            [ip] => 47.128.29.90
42
                            [network] => 47.128.29
43
                            [host] => 90
44
                        )
45
 
46
                )
47
 
48
            [updated_at] =>
49
            [urlhaus] => Array
50
                (
51
                )
52
 
53
            [urlhaus_network] => Array
54
                (
55
                )
56
 
57
        )
58
 
59
)

In the case of bots we'll try and provide ownership details, and we'll show other IP addresses associated with the same bot service.

In the last example we'll look at an IP that resolves as a bot and is also regarded as malware or malicious.

1
Array
2
(
3
    [is_malware] => 1
4
    [is_bot] => 1
5
    [status] => 200
6
    [data] => Array
7
        (
8
            [ip] => 195.191.219.130
9
            [network] => 195.191.219
10
            [bot] => Array
11
                (
12
                    [bot_name] => MJ12 Bot
13
                    [bot_category] => MJ12 Bot
14
                    [bot_url] => http://majestic12.co.uk/bot.php
15
                    [bot_producer_name] => Majestic-12
16
                    [bot_producer_url] => http://majestic12.co.uk
17
                    [bot_count] => 28
18
                    [ip] => Array
19
                        (
20
                            [0] => 158.220.119.91
21
                            [1] => 37.57.218.243
22
                            [2] => 195.191.219.130
23
                            [3] => 217.182.175.222
24
                            [4] => 195.191.219.131
25
                            [5] => 195.191.219.132
26
                            [6] => 217.76.60.62
27
                            [7] => 158.220.111.44
28
                            [8] => 195.191.219.133
29
                            [9] => 217.182.175.187
30
                            [10] => 217.76.60.60
31
                            [11] => 135.181.213.219
32
                            [12] => 217.182.175.146
33
                            [13] => 149.202.65.189
34
                            [14] => 158.220.123.226
35
                            [15] => 38.242.211.11
36
                            [16] => 217.182.134.134
37
                            [17] => 192.99.37.132
38
                            [18] => 135.181.74.243
39
                            [19] => 65.108.203.133
40
                            [20] => 217.182.134.101
41
                            [21] => 65.108.64.210
42
                            [22] => 178.151.245.174
43
                            [23] => 95.217.195.123
44
                            [24] => 94.23.7.188
45
                            [25] => 188.165.237.120
46
                            [26] => 192.99.14.19
47
                            [27] => 149.202.86.190
48
                        )
49
 
50
                )
51
 
52
            [network_count] => 4
53
            [network_ip] => Array
54
                (
55
                    [195.191.219.132] => Array
56
                        (
57
                            [ip] => 195.191.219.132
58
                            [network] => 195.191.219
59
                            [host] => 132
60
                        )
61
 
62
                    [195.191.219.130] => Array
63
                        (
64
                            [ip] => 195.191.219.130
65
                            [network] => 195.191.219
66
                            [host] => 130
67
                        )
68
 
69
                    [195.191.219.133] => Array
70
                        (
71
                            [ip] => 195.191.219.133
72
                            [network] => 195.191.219
73
                            [host] => 133
74
                        )
75
 
76
                    [195.191.219.131] => Array
77
                        (
78
                            [ip] => 195.191.219.131
79
                            [network] => 195.191.219
80
                            [host] => 131
81
                        )
82
 
83
                )
84
 
85
            [updated_at] => 1714870285
86
            [urlhaus] => Array
87
                (
88
                )
89
 
90
            [urlhaus_network] => Array
91
                (
92
                )
93
 
94
        )
95
 
96
)

  The Impact of Bots

Bots such as Amazon, ByteDance, MJ12 (Malware) and a range of others often provide no relevance or value to the every day operation or success of our website. The aggressive bots are commonly used to ingest information to train AI systems, or they're used for harvesting information that can be sold. The net impact is that these requests slow down your site, compromise website performance, and consume valuable resources. At the time of writing we're undecided how we'll handle many bots, but it's highly likely we'll simply ban all those that don't provide direct value. Many bots, such as Google, Bing, Yandex, and others, are permanently white-listed.

We've recorded tends of thousands of bots (with many others unidentified), and the number of requests per day to our systems is often in the magnitude of millions of requests. Compared against the same period just a few years ago we've seen a massive increase that we attribute to the training of various AI systems.

Bot API Endpoint: The API includes a large number of endpoints, such as those to query bots and associated IP addresses. The API will be detailed in full (over time) via our FAQ module.

Matrix API: We have our own bot that crawls only industry-specific websites (notably the finance industry), so we're aware of the needs for bots in order to gain an understanding that can be freely shared with others. However, our own BeNet bot makes requests at intervals that are unlikely to impact server performance and the data is open-sourced to the industry. We try to adhere to a practice that we consider best-practice.

  Simplified API Endpoint

As noted a couple of times, it's the is_malware key that is most relevant, and you'll likely want to minimise the returned payload if you're using the API for any real-world server-level applications. The parameter of simple=1 will force the response to return either a 1 or 0 with no other information. Error responses are empty when the simple attribute is included.

  Xena Integration

We recently overhauled the entire statistical engine into a module called Xena. The engine provides advanced statistics at the website level and extremely comprehensive statistics at the Yabber level (the former is provided for those that don't have Yabber access). Until now, we've recorded all resolved bots in a is_bot field and we've returned that information when specifically called for in Yabber tables and API responses. The number of bot records is extremely significant. As of this week you'll start to see a drastically reduced count of bot views.

  More Information

It's expected that we'll include API documentation into our FAQ module. If you're in immediate need for a solution we'd ask that you contact us for more information.

  Featured Image: Bellingen branch of the Bank of Australasia, c.1930 (Bellington is on the NSW North Coast a short drive to the south from Coffs Harbour). Originally the site of the First Catholic Church. In 1925 the Bank of Australasia (later ANZ) constructed bank premises and residence on site. Converted to shops in 1970. The Bank of Australasia took a keen interest in the Northern Rivers district in 1911 when the railway line was being extended in that direction. A special survey of likely places for branch extension was made by senior officers of that bank, and, as a result the Bank of Australasia opened at Bellingen on the 4th of August 1911; and at Dorrigo on the 1st of August 1911. The inscription on the reverse of this photograph reads as follows: "Premises occupied by the Bank of Australasia which were built in 1923 at a cost of £7,760 (including the land which cost £1,510)". The Bank of Australasia was an Australian bank in operation from 1835 to 1951. The bank merged with the Union Bank of Australia to form the Australia and New Zealand Bank on the 1st October 1951. [ View Image ]

■ ■ ■

 
Download our complimentary 650-page guide on marketing for mortgage brokers. We'll show you exactly how we generate billions in volume for our clients.
Finance Guide, Cropped Top and Bottom
  Timezone: 1 · [ CHANGE ]

RELATED READING

Like this article?

Share on Facebook
Share on Twitter
Share on Linkdin
Share on Pinterest

Leave a comment