{"id":1105,"date":"2016-03-02T15:27:40","date_gmt":"2016-03-02T23:27:40","guid":{"rendered":"https:\/\/www.cloudinsidr.com\/content\/?p=1105"},"modified":"2016-03-28T00:26:05","modified_gmt":"2016-03-28T08:26:05","slug":"how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks","status":"publish","type":"post","link":"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/","title":{"rendered":"How to Set Up Anti-Spam Filters Using Regular Expressions the Smart Way"},"content":{"rendered":"

In order to fish our spammer’s emails and other identifying information, you can use your mail clients’ log files and\/or junk email. Here is how to get started.<\/p>\n

<\/p>\n

Collect\u00a0historical data on mail processing<\/h2>\n

If you trust your users to catch at least some of the spam, you can collect spam activity logs, legitimate mail, and discarded junk mail from their mail clients to try to set up server-side mail processing\u00a0filters with high accuracy.<\/p>\n

Collecting spam activity logs from mail clients<\/h3>\n

Spam activity logs can be useful in designing blacklists to allow your mail client to identify and appropriately process\u00a0unwanted mail.<\/p>\n

For Postbox, the spam activity log is a file named junklog.html. It is conveniently located in the root of the directory that hosts the user’s Postbox profile. Use the Postbox Profile Manager to figure out the exact location of each profile and the corresponding logs. On Windows, users of Postfix can simply execute Run and enter:<\/p>\n

postbox.exe -ProfileManager<\/pre>\n
In the window that follows, Postbox will show the location of the profile in a cloud.<\/p>\n
\"Postbox<\/a>
Postbox Profile Manager: the path containing the profile leads directly to the Spam log<\/figcaption><\/figure>\n
Collecting\u00a0discarded\u00a0junk messages from email clients<\/h3>\n
Alternatively, you can collect junk\u00a0messages in a temporary directory on your local computer\u00a0in order to extract spammers’ identifying information from it. (Just make sure your internet security software is up and running.)<\/p>\n
Collecting legitimate mail to set up\u00a0whitelists<\/h3>\n
Mail clients allow you to save individual messages\u00a0(usually as separate .eml files). This way, you can extract and whitelist legitimate mail addresses to prevent Postfix from classifying them as spam.<\/p>\n
Concatenate files<\/h3>\n
In order\u00a0to concatenate files on Windows, start Command Prompt and enter:<\/p>\n
copy \/a *.html concatenated-logs.txt<\/pre>\n
The system will concatenate all files with the file extension .html (this includes Postbox’ log files in\u00a0.html) in the current directory. For .eml files (individual junk messages) use:<\/p>\n
copy \/a *.eml concatenated-messages.txt<\/pre>\n
Depending on how many files you are dealing with, it can take some time. The resulting file will be saved in the same directory.<\/p>\n
Extract spammer’s email addresses<\/h2>\n
There are several ways you can teach\u00a0Postfix to recognize and handle spam (read about types of smtpd access restrictions in Anti-Spam Defense: Using Postfix With smtpd Access Restrictions<\/a><\/em>).<\/p>\n
Valid email addresses should conform, at lest in theory, to a standardized syntax that can be verified using a regular expression. Don’t bother verifying standards compliance, though. Why should spammers be conforming to anything if all they want is to spam you? What you really want is a RegEx pattern that will tolerate weird characters such as forward slashes and underscores.<\/p>\n
In order to simplify dealing with line-end encoding, you may want to use the command\u00a0“Edit > EOL Conversion” in Notepad++ to conform it to a standard of your choice. LF is represented by \\n.\u00a0CR\/LF is represented by \\r\\n. In any case, be mindful of the appropriate line-end encoding.<\/p>\n
\"Make<\/a>
Make sure you are using appropriate line ending for your system<\/figcaption><\/figure>\n
In order to clean up the file by removing unnecessary content leaving only the relevant information in place, you need to\u00a0place each\u00a0email address on its own separate line, bookmark all relevant lines based on a regex, and remove all lines that haven’t been bookmarked.<\/p>\n
Step 1. Mark email addresses using a regular expression and inspect the\u00a0accuracy of the\u00a0results<\/h5>\n
First, you need to figure out how to capture spammers’\u00a0email addresses using a\u00a0regex that best matches\u00a0the characteristics of your input file. When performing replacements in order to separate email addresses from the remaining text, a more tolerant regular expression means a lower accuracy and a higher need for manual post-processing.<\/p>\n
WARNING:\u00a0A regular expression that works with the Mark feature in Notepad++ can fail when used in a replacement.\u00a0In order to find\u00a0email addresses in\u00a0concatenated mail messages\u00a0containing a pattern like this one:<\/p>\n
<SportUtilityVehicles-@whatever.net.in>; Sun, 15 May ... envelope-from <yadayada@whatever.tld>)<\/pre>\n
you could use roughly the following\u00a0regular expression (positive lookbehind, followed by the actual email pattern, followed by a positive lookahead):<\/p>\n
(?<=[<])(\\S*?(?=[@]))@(.+?(?=[>]))<\/pre>\n
Using a\u00a0positive lookbehind\u00a0and a positive lookahead offers you the ability to correctly mark anything resembling an email address enclosed in <> with\u00a0the Search > Mark… feature in Notepad++. You could refine the first and second capture group for more accuracy, but unfortunately,\u00a0this regular expression will fail when used in an\u00a0Replace operation\u00a0in Notepad++. For replacements, you need something other than that.<\/p>\n
Begin by\u00a0capturing From: email addresses:<\/p>\n
^(From:.+)(<)(([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\'\\\u2013\\&\\+]*)@(([A-Za-z0-9\\-\\_\\\/\\=\\.)*([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?))(>)<\/pre>\n
with this replacement (in a CR\/LF file):<\/p>\n
$1$2\\r\\n$3\\r\\n$7<\/pre>\n
in order to split From: with the sender’s mail account name from the actual email address.\u00a0When you run this replacement, Notepad++ should place each email address on its separate line.<\/p>\n
Next, you want to remove\u00a0all contents of the file except for the sender’s (spammer’s) email that’s now sitting on its own separate line. Here is how to do it.<\/p>\n
Step 2. Find and\u00a0bookmark lines that contain spammers’ email addresses<\/h5>\n
Enter this expression in Find what (in a CR\/LF file):<\/p>\n
(?<=(<\\r\\n))(([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\’\\\u2013\\&\\+]*)@(([A-Za-z0-9\\-\\_\\\/\\=\\.)*([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?))$<\/p>\n
Switch over to the Mark tab, activate option Bookmark line, Clear all marks if you haven’t already, and hit Mark All.<\/p>\n
Step 3. Inspect\u00a0the result<\/h5>\n
Inspect a couple of random bookmarks\u00a0for accuracy; the number of bookmarked lines should equal the number of replacement operations you made when adding newline characters to put the email addresses on their separate lines.<\/p>\n
Step 4. Remove everything except for bookmarked lines<\/h5>\n
Select Edit > Bookmark > Remove unselected lines.<\/p>\n
\"Spammers'<\/a>
Spammers’ emails in Notepad++: you may be tempted to block some tlds (such as .pro) completely; on the other hand, make sure you don’t capture legitimate addresses (for example your own), generic user names (info@), or domain names of legitimate email services (ymail.com)<\/figcaption><\/figure>\n
Make sure that all lines in the file containing the email addresses you collected conform to this regex (line end CR\/LF):<\/p>\n
^([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\’\\\u2013\\&\\+]*)@(([A-Za-z0-9\\-\\_\\\/\\=\\.)*([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?)(.+)\\r\\n<\/p>\n
Find email addresses in spam logs\u00a0based on a regular expression\u00a0and separate spammers’ addresses by placing each on its own line<\/h4>\n
In order to extract email addresses from the html log of Postbox that looks like this:<\/p>\n
&lt;SportUtility%Vehicles-@whatever.net.in&gt;<\/pre>\n
you could use this regular expression:<\/p>\n
([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\'\\\u2013\\&\\+]+)@(([A-Za-z0-9\\-\\_\\\/\\=)([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?)(.+)(?=&)<\/pre>\n
If you were to insist on a positive lookbehind (not supported in JavaScript), you might try this:<\/p>\n
(?<=&lt;)([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\'\\\u2013\\&\\+]+)@(([A-Za-z0-9\\-\\_\\\/\\=)([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?)(.+)(?=&)<\/pre>\n
Mark email addresses and\u00a0inspect the\u00a0accuracy of the\u00a0regular expression<\/h5>\n
To verify that the regular expression of your choice is doing what you intend it to do, open the log file in Notepad++ (or another similarly capable code editor) and use the command Search > Mark… to open the Mark dialog. In Find what:, enter:<\/p>\n
(([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\’\\\u2013\\&\\+]+)@(([A-Za-z0-9\\-\\_\\\/\\=)([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?)(.+))(?=&)<\/p>\n
activate the Bookmark line option, then hit Mark All. Notepad++ will highlight all occurrences of a text string that conforms to the pattern you specified. Inspect it as thoroughly as it appears reasonable, to ensure that the regular expression is doing a decent job of finding spammers’ addresses without capturing the recipients’.<\/p>\n
Separate spammers’ addresses by placing each on its own line<\/h5>\n
In order to separate spammer’s email addresses from the rest of the document so that each sits on its own separate line, switch to the Replace tab in Notepad++. In Replace with, enter:<\/p>\n
\\n$1\\n<\/pre>\n
and hit Replace All in order to add a line break both before and after each occurrence of the first capture group. (Clear all marks, if necessary.)<\/p>\n
Mark spammers’ email addresses with the Bookmark lines option activated to define lines to\u00a0retain<\/h5>\n
In Notepad++, use the Mark feature with the option Bookmark lines activated to highlight the following pattern:<\/p>\n
^([A-Za-z0-9\\-\\_\\\/\\.\\=\\:\\%\\!\\'\\\u2013\\&\\+]+)@(([A-Za-z0-9\\-\\_\\\/\\=)([A-Za-z0-9\\-\\_\\\/\\=]+)\\.?)(.+)\\n<\/pre>\n
(in a file with an LF line end, \\n will cause line break as it represents the newline character; in files with CRLF line end encoding, use \\r\\n instead, where \\r represents carriage return and \\n a new line).<\/p>\n
Remove unmarked lines and clean up the results<\/h4>\n
Using the command Search > Bookmark > Remove Unmarked lines<\/strong>, clear the document of all noise, leaving–hopefully–only spammers’ emails in place. Save the file, then inspect it again carefully. When you notice something amiss, make the necessary corrections. For example, you may want to search for a double forward slash (as in imap:\/\/), a mailto:, @font-face, @-ms-viewport\u00a0or a space, and see if any unusual remnants of the original log file come up. If they do, see whether the “offending”\u00a0lines contain\u00a0duplicates of information you already have; if they do, you could simply mark these lines and remove them (it’s usually easier than cleaning them up or further tweaking your original regex). All of it depends how much data you are dealing with.<\/p>\n
The art of whitelisting: permit\u00a0legitimate email addresses<\/h2>\n
Probably the\u00a0easiest way to do it involves concatenating\u00a0emails that are known to be legitimate, capturing domain names from email addresses, and whitelisting them with the OK directive, with regex:<\/p>\n
\/.*\\.*client1\\.tld$\/ \u00a0 \u00a0 OK<\/pre>\n
This will permit any email address and any hostname associated with the domain client.tld to pass the checks.<\/p>\n
Block the biggest offenders on a top-level domain basis (yes, it’s cruel)<\/h2>\n
This may sound controversial and it is indeed a sign of desperation, but after sifting through millions of lines containing email addresses of spammers you may come to the same conclusion: spammers\u00a0like some top level domains more than others, and they seem to be alone in their passion for them. We have yet to see a legitimate user of a domain\u00a0that ends in .xyz or .date. Depending on your business, your mileage may vary, but if your particular situation allows you to block spammy senders on a\u00a0tld level, then by all means, do it. We would caution you, however, from blocking country specific tlds such as .ru or .co, regardless of how spammy they are,\u00a0as a matter of fairness towards legitimate users in those geographic locations.<\/p>\n
In order to block a tld using a regex, use this syntax in your access maps with either REJECT (and an optional message that gets delivered with the bounce) or DISCARD (and an optional note to yourself that gets logged by\u00a0your mail server):<\/p>\n
\/\\.usa\\.cc$\/ \u00a0 \u00a0 REJECT\r\n\/\\.xyz$\/ \u00a0 \u00a0 REJECT<\/pre>\n
Once this general block is in place, you no longer need to bother sifting through email addresses that use these domain names; when you discard them from your collection of spammers’ emails you get to focus on those that cannot be blocked this way (such as, most notably, .com).<\/p>\n
Bookmark all lines that you no longer need using a regular\u00a0expression like this one:<\/p>\n
(click|bid|club|date|download|faith|link|pro|racing|review|science|space|top|usa.cc|website|work|xyz)$<\/pre>\n
and remove all bookmarked lines from the file. This leaves you with plenty of other tlds to deal with.<\/p>\n
Filter for spammy email usernames and remove duplicates<\/h2>\n
Sort collected\u00a0email addresses\u00a0“Edit > Line operations > Sort lines […]” and write regex for user names. Once you have entered offensive user names\u00a0into your configuration file you can strip them\u00a0from the email addresses (username@) in your work file in Notepad++.<\/p>\n
To remove\u00a0user names from email addresses leaving only hostnames in place, search for:<\/p>\n
(.+)@(.+)<\/pre>\n
and replace with:<\/p>\n
$2<\/pre>\n
Extract\u00a0domains with known two-part top level domain extensions, for example:\u00a0in.net, co.za or co.uk, and save them in a separate file to\u00a0process manually; otherwise you would block all domains with these extensions!\u00a0Mark all lines containing the pattern:<\/p>\n
(.+)\\.(.+)\\.(.+)<\/p>\n
and remove host names with a tld that contains a dot (co.uk or com.br or co.in). In the remaning file, strip the host names using this replacement pattern:<\/p>\n
$2\\.$3<\/p>\n
Remove duplicates by selecting the contents of your file (Ctrl-A) and using the command “TextFX > TextFX Tools > Sort lines case insensitive (at column)” with option “Sort outputs only UNIQUE (at column)” activated (TextFX is an extension to Notepad++).<\/p>\n
It is time for you to turn your attention to the domain names.<\/p>\n
Step 6. Filter offending domain names<\/h3>\n
In order to capture future spam from domains that have spammed you before, allow for any\u00a0hostname on those domains to be rejected.\u00a0This way, whoever wants to spam you again needs a new domain name and that means additional\u00a0expenses.<\/p>\n
In Find what, enter:<\/p>\n
^([A-Za-z0-9\\-]*\\.)*([A-Za-z0-9\\-]+)\\.([a-z]+)$<\/pre>\n
In Replace with, enter:<\/p>\n
\/.*\\\\.*$2\\\\.$3\\$\/ \u00a0 \u00a0 DISCARD sender rejected<\/pre>\n
turns host.somedomain.tld to<\/p>\n
\/.*\\.*subdomain\\.tld$\/ \u00a0 \u00a0DISCARD sender rejected<\/pre>\n
Perform another search and replace to mask every – as \\- (use the Normal mode for this operation).<\/p>\n
Step 7. Trust but verify<\/h3>\n
To verify how your rules are being applied, you can run this test on the command line of your server (with the path to the\u00a0list of access restrictions you want to test):<\/p>\n
postmap -q \"teststring\" regexp:\/etc\/postfix\/access_maps\/regex_access_sender<\/pre>\n
Now you also have to make sure that Postfix actually reads this list. Here is how to add it to your configuration of Postfix smtpd restrictions<\/a>.<\/p>\n
When you are all done, have a quick look at current\u00a0log activity:<\/p>\n
tail -f \/var\/log\/maillog<\/pre>\n","protected":false},"excerpt":{"rendered":"
In order to fish our spammer’s emails and other identifying information, you can use your mail clients’ log files and\/or junk email. Here is how to get started.<\/p>\n","protected":false},"author":101011,"featured_media":76,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","footnotes":""},"categories":[2],"tags":[29],"yoast_head":"\nHow to Set Up Anti-Spam Filters Using Regular Expressions the Smart Way - CloudInsidr<\/title>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Set Up Anti-Spam Filters Using Regular Expressions the Smart Way - CloudInsidr\" \/>\n<meta property=\"og:description\" content=\"In order to fish our spammer’s emails and other identifying information, you can use your mail clients’ log files and\/or junk email. Here is how to get started.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/\" \/>\n<meta property=\"og:site_name\" content=\"CloudInsidr\" \/>\n<meta property=\"article:published_time\" content=\"2016-03-02T23:27:40+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-03-28T08:26:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.cloudinsidr.com\/content\/wp-content\/uploads\/2015\/11\/cloudinsidr_logo_900px-wide.png\" \/>\n\t<meta property=\"og:image:width\" content=\"900\" \/>\n\t<meta property=\"og:image:height\" content=\"326\" \/>\n<meta name=\"twitter:card\" content=\"summary\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#website\",\"url\":\"https:\/\/www.cloudinsidr.com\/content\/\",\"name\":\"CloudInsidr\",\"description\":\"Cyber security, infotech\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/www.cloudinsidr.com\/content\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/www.cloudinsidr.com\/content\/wp-content\/uploads\/2015\/11\/cloudinsidr_logo_900px-wide.png\",\"width\":900,\"height\":326,\"caption\":\"cloudinsidr.com logo (900px wide)\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/#webpage\",\"url\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/\",\"name\":\"How to Set Up Anti-Spam Filters Using Regular Expressions the Smart Way - CloudInsidr\",\"isPartOf\":{\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/#primaryimage\"},\"datePublished\":\"2016-03-02T23:27:40+00:00\",\"dateModified\":\"2016-03-28T08:26:05+00:00\",\"author\":{\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#\/schema\/person\/21ce63bea726ea64da1beed97e63ba84\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.cloudinsidr.com\/content\/how-to-extract-spammers-identifying-information-from-logs-and-junk-emails-using-regex-and-other-tricks\/\"]}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#\/schema\/person\/21ce63bea726ea64da1beed97e63ba84\",\"name\":\"Anna E Kobylinska\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/www.cloudinsidr.com\/content\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7a3e9bd152f9d5cd41bf2b92df649857?s=96&d=mm&r=g\",\"caption\":\"Anna E Kobylinska\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts\/1105"}],"collection":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/users\/101011"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/comments?post=1105"}],"version-history":[{"count":52,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts\/1105\/revisions"}],"predecessor-version":[{"id":1180,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/posts\/1105\/revisions\/1180"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/media\/76"}],"wp:attachment":[{"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/media?parent=1105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/categories?post=1105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cloudinsidr.com\/content\/wp-json\/wp\/v2\/tags?post=1105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}