remove duplicate from text file based upon column

-

i have text file has format , ends each line crlf. want remove dups file. found few examples seemed double file size when creating new file without dups and
seemed usable. take file use sql(dts) load text file table. want scrub text file before loading.

 

"test1"|"455"|"34166"|"06/23/11"
"test2"|"450"|"34167"|"06/23/11"
"test2"|"450"|"34167"|"06/23/11"

 thanks.

 

add out-file.

gc file.txt | sort -uniqu | out-file result.txt

or

gc file.txt | sort {$_.split("|")[2]} -unique | out-file result.txt



Windows Server  >  Windows PowerShell



Comments

Post a Comment

Popular posts from this blog

some help on Event 540

-
does event 540 me distinguish between user , administrator access?    this event generated ypon successfull logon, logon exactly? successful logon on system?   is there other way log administrator access system using event viewer?   thank valuable response   georges hi,   as far know, cannot distinguish between user , administrator access event 540. more information, please refer to:   event id: 540 http://www.microsoft.com/technet/support/ee/transform.aspx?prodname=windows+operating+system&prodver=5.0&evtid=540&evtsrc=security&lcid=1033   events 528 , 540 http://blogs.msdn.com/b/ericfitz/archive/2004/12/09/279282.aspx   regards, bruce this posting provided "as is" no warranties, , confers no rights. please remember click "mark answer" on post helps you, , click "unmark answer" if marked post not answer question. can beneficial other community members reading thread.
Read more

WMI Repository 4GB limit - Win 2003 Ent Question

-
Image
hi there, i've question size limit on wmi repository (being 4gb). if isn't correct place post this, apologies , can let me know wheere post this? i've come across few servers running citrix have reached limit on wmi repository size , need install ibm director agent hw monitoring. during install getting 'out of disk space' errors entered in mofcomp.log: "(mon apr 02 04:10:36 2012.172843) : error occurred while processing item 4: (mon apr 02 04:10:36 2012.172843) : error number: 0x8004103b, facility: wmi description: out of disk space" when error number http://msdn.microsoft.com/en-us/library/windows/desktop/aa394559%28v=vs.85%29.aspx i see this: wbem_e_out_of_disk_space 2147749947 (0x8004103b) disk out of space or 4 gb limit on wmi repository (cim repository) size reached. windows xp , windows 2000/nt:   disk out of space. i'm wondering if there reason why limit set 4gb or can increase needs be? can't seem find reas
Read more

Event ID 1302 (error 1307) DFS replication service encountered an error while writing to the debug log file

-
Image
hello. we @ step 0 of migration frs dfsr sysvol replication on windows 2008r2 dc. every time run powershell dfsrmig / getmigrationstate says: "unable create dfsr migration log file. error 1307. all domain controllers have migrated global state 'start'. migration has reached consistent state on domain controllers. succeeded." it creates event id 1302 in "dfs replication" event log, explaining error 1307 means: "this security id may not assigned ownder of object ". i already checked space limits, quotas, permissions on c:\windows\debug according microsoft support article unable fix it. although, there new entries added dfsr00095.txt log file despite error. dfsrxxxx file has not been archived .gz format, and, beleive, contains dfsr diagnostics , events. here of latest entries: 20150908 08:47:27.867 11616 sysm  3354 migration::sysvolmigration::getsysvolreadyflag [mig] sysvol ready 20150908 08:47:27.867 11616 sysm   456 migrati
Read more