How to split a huge size log file and only get what you want in Linux?

Today I encounter a task as follow:

I need to extract a 7.3GB Apache access log and ONLY grab the access log starting from 30 August and up to today.

I don't need to say it is non-sense to open a 7.3 GB file in a text editor, even in vi under linux, the RAM that is used is huge, and it just doesn't work that way.

So to solve this problem, here are the steps I took, thanks to the following reference:

- http://stackoverflow.com/questions/3066948/how-to-file-split-at-a-line-number


  1. Find out the exact string first occurrence in the log file and print the first 5 lines.  Actually I can only print one line but I just like 5:
    1. grep -n "30/Aug/2015" access_log | head -n 5
  2. The returned line will be:
    1. 61445828:203.129.95.51 - - [30/Aug/2015:00:00:01 +0800] "GET <somewebsite>/index.htm HTTP/1.1" 200 10824
    2. The first item: 61445828 is the line number
  3. Count the total number of lines in access log, and get the number:
    1. wc -l access_log
  4. The return is: 64328208 access_log.old
  5. Now, do this calculation: (Total line of access_log - Starting line of required text) = Starting line, so we just need the log starting from: 64328208 - 61445828 = 2882380.  2882380 is the starting line number
  6. We export the content starting from 2282380 to a new file: 
    1. tail -n 2882380 access_log > custom_log_file_for_analysis.log
Now that file is just 414 MB, which is 95% smaller than the original file.  

You can now do whatever analysis you want with this file.

Hope it helps someone!

Comments

Post a Comment

Popular posts from this blog

TCPDF How to show/display Chinese Character?

Thanks to this article (in Simplified Chinese, updated new link), there is a way to embed a TTF that can display Chinese (Traditional and Simplify, and possibly Japanese). Environment:  TCPDF 6.0.020 PHP 5.3.8 Windows 7 The problem: If you follow this post that said: using cid0cs or cid0ct to display Chinese, it can display the character correctly, only to the extend that user installed a correct font package.  However, chances are other users who didn't install that package cannot view that PDF since there are several packages and versions of PDF viewer (mainly Adobe).  And even worse: even if user installed certain CJK font package, the PDF still not displayed correctly (show empty page). And according to the official TCPDF documentation about font : The fonts that could be not embedded are only the standard core fonts and CID-0 fonts . The solution: Again, thanks to the post above (in simplified Chinese), now developer can use a font that: Can disp...
Read more

How to fix fancy box/Easy Fancybox scroll not work in mobile

Hi The Problem: If you are working on EasyFancybox or just normal fancybox JS , and you have this problem: In your iOS device when a popup is opened, if the page is too wide and your pop up cannot show all, when user taps and drag the screen, the background will be dragged instead of the pop-up content. To fix, add this in your CSS: -webkit-overflow-scrolling: touch !important; So which element should you apply it to? In Easy Fancybox, you will see these 2 lines: <div id="fancybox-content"........>         <iframe id="fancybox-frame" name............... Apply to fancybox-content, not fancybox-frame So in your CSS, it should be: #fancybox-content{         overflow: scroll !important;         -webkit-overflow-scrolling: touch !important; } It works for me, and I hope it helps someone also.
Read more

Wordpress Load balancing: 2 web servers 1 MySQL without any Cloud services

Image
This is the exact config that I need to configure a load balancing without other cloud services like Amazon, rackspace or GoGrid, completely customized. I put my question on stackoverflow.com but no luck, so I put it on Quora.com and immediately I got response: My Notes: The load balancer is done by my colleagues so I am not sure what is the config of it.  But one thing to note: Your LB should enable "session stickiness" so that the session will follow visitor.  By the way, I am using Round-robins. Session: Actually I did not use session, but it seems that other plugins (like WPML) will use it.  If I use Apache load balancer, when I go to wp-admin through load balancing IP, I will not able to login, given the username and password is the same.  I will be redirected to login page.  So ask your load balancer providers or check your load balancer config and see how to manage the "Session Stickiness" issue. Shared folder: your wp-content/uploads folder...
Read more