remove duplicate from text file based upon column

-

i have text file has format , ends each line crlf. want remove dups file. found few examples seemed double file size when creating new file without dups and
seemed usable. take file use sql(dts) load text file table. want scrub text file before loading.

 

"test1"|"455"|"34166"|"06/23/11"
"test2"|"450"|"34167"|"06/23/11"
"test2"|"450"|"34167"|"06/23/11"

 thanks.

 

add out-file.

gc file.txt | sort -uniqu | out-file result.txt

or

gc file.txt | sort {$_.split("|")[2]} -unique | out-file result.txt



Windows Server  >  Windows PowerShell



Comments

Post a Comment

Popular posts from this blog

Round Robin is killing performance on our network

-
Image
we had 2 windows 2003 domain controllers @ our data center dns , 1 domain controller @ our large remote office no dns. finished replacing 3 domain controllers windows 2012 r2 dc's, each of have dns installed , configured. we're having issues servers @ our data center resolving names using dns server @ remote office due appears round robin dns lookups. is there suggested way force servers , workstations @ data center use domain controllers @ location dns resolution , servers , workstations @ remote office use domain controller @ location dns resolution? thanks in advance assistance, dan hi dbetanco, >we're having issues servers @ our data center resolving names using dns server @ remote office due appears round robin dns lookups. round robin specific record query, example, in dns zone has 3 records: www.test.com 1.1.1.1 www.test.com 2.2.2.2 www.test.com 3.3.3.3 then if query nslookup www.test.com  three times, first time order is: 1.1....
Read more

WMI Repository 4GB limit - Win 2003 Ent Question

-
Image
hi there, i've question size limit on wmi repository (being 4gb). if isn't correct place post this, apologies , can let me know wheere post this? i've come across few servers running citrix have reached limit on wmi repository size , need install ibm director agent hw monitoring. during install getting 'out of disk space' errors entered in mofcomp.log: "(mon apr 02 04:10:36 2012.172843) : error occurred while processing item 4: (mon apr 02 04:10:36 2012.172843) : error number: 0x8004103b, facility: wmi description: out of disk space" when error number http://msdn.microsoft.com/en-us/library/windows/desktop/aa394559%28v=vs.85%29.aspx i see this: wbem_e_out_of_disk_space 2147749947 (0x8004103b) disk out of space or 4 gb limit on wmi repository (cim repository) size reached. windows xp , windows 2000/nt:   disk out of space. i'm wondering if there reason why limit set 4gb or can increase needs be? can't seem find reas...
Read more

Change home folder default permission?

-
i have searched on , can't seem find similar trying do. using windows server 2003 active directory. when create new user account on domain, use profile tab specify home directory user. when user logs in first time, active directory creates folder in specified path. working correctly. problem gives user full control on folder. has caused lot of problems because "clever" users have discovered thay can change permission on folder or subfolders beneath it. because don't want staff looking @ "sensitive" files. of course doesn't work since can take ownership of file , give myself permissions. causes trouble failed backups, inability scan these folders viruses, migration new storage etc. on occasion 1 of these users call me , ask file or folder restored, , @ point discover not have backup of files , cannot restore them. give them lecture, have on 2500 users, , can't talk them all. right have create account , login teh folder created, go take away full contr...
Read more