Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

In Powershell, how to read and get as fast as possible the last line (or all the lines) which contains a specific string in a huge text file (about 200000 lines / 30 MBytes) ? I'm using :

get-content myfile.txt | select-string -pattern "my_string" -encoding ASCII | select -last 1
But it's very very long (about 16-18 seconds).
I did tests without the last pipe "select -last 1", but it's the same time.
Is there a faster way to get the last occurence (or all occurences) of a specific string in huge file?
Perhaps it's the needed time ...
Or it there any possiblity to read the file faster from the end as I want the last occurence?
Thanks
                The reason there was no change whether you piped to "Select -last 1" or not is because the whole file has to be processed to know which is "last".
– Kevin Buchan
                Jan 23, 2014 at 14:17
                You may need to use .NET to have some performance there: Start reading massive text file from the end.
– Victor Zakharov
                Jan 23, 2014 at 16:11
                Even with -ReadCount=1000, get-content still reads the entire into memory.  I ran out of memory trying to parse a 40GB file.  Any other ideas?
– BlueRaja - Danny Pflughoeft
                Mar 16, 2018 at 0:24
    while (!$reader.EndOfStream) {
        $line = $reader.ReadLine()
        if ($line.Contains("my_string")) {
            $lines += $line
$lines | Select-Object -Last 1
                Hi @riabovil, what if i have two or more strings to search in a file (my_string, my_string1)?
– Naga
                Apr 22, 2020 at 6:20
Have you tried using [System.IO.File]::ReadAllLines();? This method is more "raw" than the PowerShell-esque method, since we're plugging directly into the Microsoft .NET Framework types.
$Lines = [System.IO.File]::ReadAllLines();
[Regex]::Matches($Lines, 'my_string_pattern');
                this user specifically said hes using large files, why would you post a solution that "can crash if used with large files"?
– Chad Baxter
                Nov 8, 2015 at 22:32
                if i want regex to give whole line where pattern matches how to do that [Regex]::Matches($line, 'Database:'); it should give where it matches Database: but it should give databasename as well
– deepti
                Feb 14, 2018 at 5:28
I wanted to extract the lines that contained  failed and also write this lines to a new file, I will add the full command for this
get-content log.txt -ReadCount 1000 |
>>  foreach { $_ -match "failed" } | Out-File C:\failes.txt 
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.