Monday, July 9, 2018

Split Large Files into Parts

Hi All,

For one of the Client we had started the logging of SFTP communications done through SFTP Client tool WinScp, on weekends client processing team had process many files and unfortunitely one of the file was not uploaded properly to client SFTP site.

Luckly we had enabled logging on our SFTP client, but now the challenge is over the weekend this log file is increased in MBs and it is difficult to read such a hugh file with Windows Notepad or even NotePad++ 

So i thought to develop a small utility to split such large file into parts so that it can be access in parts to read the contents.

I wrote the following code in C# but i am pretty sure it can be convert in any other native languages supported by different operating systems platforms.

Here is the code snippet of the utility followed by usage and results screenshot.

namespace ServiceConsoleApp
{
    class Program
    {

        static void Main(string[] args)
        {
           
            var fileSuffix = 0;
            int lines = 0;
            Stream fstream = File.OpenWrite(args[1] + (++fileSuffix));
            StreamWriter sw = new StreamWriter(fstream);

            using (var file = File.OpenRead(args[0]))
            using (var reader = new StreamReader(file))
            {
                while (!reader.EndOfStream)
                {
                    sw.WriteLine(reader.ReadLine());
                    lines++;

                    if (lines >= 1000000)
                    {
                        sw.Close();
                        fstream.Close();
                        lines = 0;
                        fstream = File.OpenWrite(args[1] + (++fileSuffix));
                        sw = new StreamWriter(fstream);
                    }
                }
            }

            sw.Close();
            fstream.Close();
        }
    }
}

Syntax:

YourDrive:\>ServiceConsoleApp.exe FTPLog.log FileSplits



Hope this Helps!!!!!!