find all the ascii files with certain pattern - the fastest way

find . -type f -exec grep -Iq 'UseConcMarkSweep' {} \; -ls

find and the -printf argument formats in relation with xargs -n

Directives
%k size in 1K , %b in 512-byte blocks , %s in bytes. 
%d depth in the directory tree; 0 i.e. the file is a command line argument. 
%h Leading directories of file's name (all but the last element). 
%f name with any leading directories removed (only the last element). 
%p name. 
%P File's name with the name of the command line argument under which it was found removed. 
%H Command line argument under which file was found. 
%F Type of the filesystem the file is on; this value can be used for -fstype. 
%g group name, or numeric group ID if the group has no name. 
%G numeric group ID. 
%i inode number 
%l Object of symbolic link (empty string if not a symbolic link). 
%m permission bits (in octal). 
%n Number of hard links to file. 
%u owner, or numeric user ID if the user has no name. %U numeric user ID. 
A % character followed by any other character is discarded (the other character is output).

Eclipse: edit locally and automatic rsync as part of the build

So I have this project that I love to edit, but need to build and deploy on a different target system (in my case OSX to Linux) I also love to edit in Eclipse, and use rsync as the fastest way to get stuff on the target machine. The remote systems explorer just did not cut it... however I ended up flipping between eclipse and terminal (rsync) to get things synced up.... Answer: add a builder launcher that invokes the rsync script as part of the autobuild... Here the simple rsync script that I placed in root of the maintenance-ui project (chmod it to 755)

rsync -avz --exclude '.svn' --exclude 'target' --delete *  myuserid@myhost.com:./projects/maintenance-ui

Here the launch file

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<launchConfiguration type="org.eclipse.ui.externaltools.ProgramBuilderLaunchConfigurationType">
<booleanAttribute key="org.eclipse.debug.ui.ATTR_LAUNCH_IN_BACKGROUND" value="true"/>
<stringAttribute key="org.eclipse.ui.externaltools.ATTR_LOCATION" value="${workspace_loc:/maintenance-ui/rsync.sh}"/>
<stringAttribute key="org.eclipse.ui.externaltools.ATTR_RUN_BUILD_KINDS" value="full,auto,"/>
<booleanAttribute key="org.eclipse.ui.externaltools.ATTR_TRIGGERS_CONFIGURED" value="true"/>
<stringAttribute key="org.eclipse.ui.externaltools.ATTR_WORKING_DIRECTORY" value="${workspace_loc:/maintenance-ui}"/>
</launchConfiguration>

Now you could and can use the remote systems explorer in Eclipse to run your remote build and you will never have to leave Eclipse and alt tabbing around.

Dumping Environment process info before an exit at startup time (JVM)

I have this process that craps out at startup time and I'm not sure what the problem is and what the exact environment settings are at runtime startup. So I need to trap the information from the /proc/[pid]/environ file on linux. In session one I start the monitoring process that will dump the environment to a file. In session two I start the actual process..... once the process startsup the session 1 oneliner will trap the information for me.

  628  while (true) do PID=`/sbin/pidof studio-tools`;if [ -n "$PID" ]; then cat /proc/$PID/environ | tr \\0 \\n > out; /usr/sbin/lsof -p $PID >> out ; break; fi; done;

rpcinfo to process information linux

The trickiness of NFS tracing and locking.... wonderful world...

# find the nfs mount
$ mount 
nas0964p:/prod_nas on /nas/images type nfs (rw,nosuid,soft,intr,tcp,addr=10.75.129.131)
$ rpcinfo -p 10.75.129.131
   program vers proto   port
 824395111    1   tcp  49394
 824395111    1   udp  52370
    100011    1   tcp  60884  rquotad
    100011    1   udp  62423  rquotad
 536870914    1   udp   4658
 536870914    1   tcp   4658
    100021    3   udp  58855  nlockmgr
    100021    2   udp  58855  nlockmgr
    100021    1   udp  58855  nlockmgr
    100021    4   udp  58855  nlockmgr
    100021    3   tcp  63916  nlockmgr
    100021    2   tcp  63916  nlockmgr
    100021    1   tcp  63916  nlockmgr
    100021    4   tcp  63916  nlockmgr
    100024    1   udp  54825  status
    100024    1   tcp  54825  status
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    140391    1   udp  31491
    100005    3   tcp   1234  mountd
    100005    2   tcp   1234  mountd
    100005    1   tcp   1234  mountd
    100005    3   udp   1234  mountd
    100005    2   udp   1234  mountd
    100005    1   udp   1234  mountd
 536870919    3   tcp  12345
 536870919    1   tcp  12345
 536870919    3   udp  12345
 536870919    1   udp  12345
    102660    1   udp  62703
    102660    1   tcp  61254
    100000    2   udp    111  portmapper
    100000    2   tcp    111  portmapper
# what is the program???
$ $ cat /etc/rpc
#ident "@(#)rpc 1.11 95/07/14 SMI" /* SVr4.0 1.2 */
#
# rpc
#
portmapper 100000 portmap sunrpc rpcbind
rstatd  100001 rstat rup perfmeter rstat_svc
rusersd  100002 rusers
nfs  100003 nfsprog
ypserv  100004 ypprog
mountd  100005 mount showmount
ypbind  100007
walld  100008 rwall shutdown
yppasswdd 100009 yppasswd
etherstatd 100010 etherstat
rquotad  100011 rquotaprog quota rquota
sprayd  100012 spray
3270_mapper 100013
rje_mapper 100014
selection_svc 100015 selnsvc
database_svc 100016
rexd  100017 rex
alis  100018
sched  100019
llockmgr 100020
nlockmgr 100021
x25.inr  100022
statmon  100023
status  100024
bootparam 100026
ypupdated 100028 ypupdate
keyserv  100029 keyserver
sunlink_mapper 100033
tfsd  100037
nsed  100038
nsemntd  100039
showfhd  100043 showfh
ioadmd  100055 rpc.ioadmd
NETlicense 100062
sunisamd 100065
debug_svc  100066  dbsrv
ypxfrd  100069  rpc.ypxfrd
bugtraqd 100071
kerbd  100078
event  100101 na.event # SunNet Manager
logger  100102 na.logger # SunNet Manager
sync  100104 na.sync
hostperf 100107 na.hostperf
activity 100109 na.activity # SunNet Manager
hostmem  100112 na.hostmem
sample  100113 na.sample
x25  100114 na.x25
ping  100115 na.ping
rpcnfs  100116 na.rpcnfs
hostif  100117 na.hostif
etherif  100118 na.etherif
iproutes 100120 na.iproutes
layers  100121 na.layers
snmp  100122 na.snmp snmp-cmc snmp-synoptics snmp-unisys snmp-utk
traffic  100123 na.traffic
nfs_acl  100227
sadmind  100232
nisd  100300 rpc.nisd
nispasswd 100303 rpc.nispasswdd
ufsd  100233 ufsd
pcnfsd  150001 pcnfs
amd  300019  amq
sgi_fam  391002 fam
bwnfsd  545580417
fypxfrd  600100069 freebsd-ypxfrd
$ 

using find and report on several lines with grep like functionality

Oneliner for finding a specific release of an artifact in a maven build file and reporting the artifact name and version even though these pieces are in different lines... And using find with the exit to print out only the successful files.

$ find . -name pom.xml -type f -exec perl -ne 'BEGIN {my $l} if (/2.0.1.RELEASE/) { printf "%s%s",$last,$_ ; $l=$last } ; $last=$_; END { exit 1 unless  $l; }' {} \; -print
   fmon-jms
   2.0.1.RELEASE
./master-service/master-service/master-proc/master-proc-core/pom.xml

Whats the encoding of my file? Compile complaints....

on Linux use... file -bi option

[cpcdev@dpcpca01 cpc]$ file -bi ./src/main/java/com/fedex/ground/dispatch/cpcs/actions/OnCallDetailActionHandler.java
text/x-java; charset=unknown
[cpcdev@dpcpca01 cpc]$ 

Or in Mac OSX

file -I --mime-encoding /prod/PersistanceLayer/src/java/domain/server/toplink/Equipment.java

Class Magic Number And Bytecode compatibility on Linux

Based on the general layout (see item #7)we need the 8th hex number to get the value. And then match the major version number with the JDK version compatibility.

major version number of the class file format being used.
J2SE 7 = 51 (0x33 hex),
J2SE 6.0 = 50 (0x32 hex),
J2SE 5.0 = 49 (0x31 hex),
JDK 1.4 = 48 (0x30 hex),
JDK 1.3 = 47 (0x2F hex),
JDK 1.2 = 46 (0x2E hex),
JDK 1.1 = 45 (0x2D hex).

In this case

[me@hostandre]$ xxd -p  org/apache/xmlbeans/impl/xsd2inst/SampleXmlUtil.class  | head -n 1 | cut -c-16
cafebabe00000030
[me@hostandre]$

The answer is: JDK 1.4 compiled and compatible...

Zip up my logfiles between 10 and 15 days old from a remote host...

Sledgehammering the logs archiving for perticular days and using zip... Tar always gives me a headache because of the -c vs -u option... why can't it just create the tar... anyway with zip its smoother and "smaller"!!

$ cat list  | xargs -i% -t ssh % 'find ./logs/ -mtime +10 -mtime -15 -exec zip -@ /tmp/my.zip "{}" \; -print '

Where list contains

me@host1
me@host2
me@host3

pidof and /proc/pid/environ values and ps and ...

All this time and I was not aware

/sbin/pidof java | sed 's/\s/\n/g'

cat /proc/%/environ | tr \\0 \\n
combined.... 
/sbin/pidof java | sed 's/\s/\n/g' | 

xargs -i% cat /proc/%/environ | tr \\0 \\n
xargs -i% -t ps -o uid,pid,start_time,ucomm -p % w 2> /dev/null  
xargs -i% -t ps -o uid,pid,start_time,comm,args -p % wwe ' 


/sbin/pidof java | sed 's/\s/\n/g' |xargs -i% -t ps -o uid,pid,start_time,comm,args -p % wwe ' 

How do I remove filenames with dashes?

This does not work....

ITASC $ rm '-z'
rm: illegal option -- z
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file

Simple: use the /- to depict dashes:

./CustomFunctions/.classpath
./-z
tibco@pghdbeengl01:~/build
$ rm ./-*

Mac OSX - running xterm and X11 xhost: unable to open display

So alright you have Mac OSX. Yay I I have a VPN to the office. Yay II (ip vpn adapter: 172.18.146.250) I have oodles of linux boxes! Yay III The problem: One odd ball process on my linux will throw a X11 Graphics error so how to get around that.... Try 1: Do the standard export and xhost + trick open Terminal.app and run:

computer:~ ~$ export DISPLAY=172.18.146.250:0
computer:~ ~$ xhost +
xhost:  unable to open display "172.18.146.250:0"
computer:~ ~$ 

Thats not right... lets start the Xserver in background (X Server startcmd is 'X').... and do this again...

computer:~ ~$ X &
launch_msg("CheckIn") IPC failure: Operation not permitted
Xquartz: Unable to locate waiting server: org.x.X11
Xquartz: X11.app = /Applications/Utilities/X11.app/Contents/MacOS/X11
Xquartz: Starting X server: /Applications/Utilities/X11.app/Contents/MacOS/X11 --listenonly
..SCM BUILD BOX profile been setup..
================================================
X11.app: main(): argc=2
 argv[0] = /Applications/Utilities/X11.app/Contents/MacOS/X11.bin
 argv[1] = --listenonly
Waiting for startup parameters via Mach IPC.
X11.app: No launchd socket handed off, unsetting DISPLAY
X11.app: do_start_x11_server(): argc=1
 argv[0] = X
Xquartz starting:
X.Org X Server 1.4.2-apple56
Build Date: 20100624
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base
(EE) XKB: Couldn't open rules file /usr/X11/share/X11/xkb/rules/base

computer:~ ~$

Try 2: Now open the xterm under the X11 Application that has been started...

bash-3.2$ export DISPLAY=172.18.146.250:0
bash-3.2$ xhost +
access control disabled, clients can connect from any host
bash-3.2$ 

Where Are My Open Files Go? lsof++

So where are my IO handles (open files) are being spend... here a oneliner First get the PID and ram everything of lsof thru the parser....

[wev@wad03 logs]$ /usr/sbin/lsof -p 21336 | perl -MData::Dumper -ne 'BEGIN { my %x=();} {@s=split /\s+/;$x{$s[4]}++;} END { print Dumper(\%x); }'
$VAR1 = {
          'IPv6' => 3,
          'unix' => 1,
          'DIR' => 2,
          'TYPE' => 1,
          'CHR' => 4,
          '0000' => 1,
          'REG' => 463,
          'sock' => 1
        };
[wev@wad03 logs]$ 

lsof: poor mans list open files

list open files per process


ps aux | grep weblogic.Name | grep -v grep | perl -ne '@x=split;my(@c)=glob("/proc/$x[1]/fd/*");printf "%5d %s open files.\n",$x[1],scalar @c;'


Reading the wtmp files in /var/log

$ utmpdump /var/log/wtmp

move extension on files

perl -e 'foreach my $r (glob("*.jad")) { $x=$r;$r=~ s/jad/java/;system("mv $x $r") unless(-f $r)}'

How to download ASL wt DVD script

#!/usr/bin/perl 

my $year = 2010;
my $x = 0;
for my $month ( 1 .. 12 ) {
   printf "downloading month %s", $month;
   for my $article ( 1 .. 6 ) {
       $file = sprintf "%s_%s_%04d%02d%02d_%02d.m4v", "w", "ASL", $year, $month, 15, $article;
       next if -e $file and -s $file < 16;
       $url = sprintf "http://download.jw.org/files/media_magazines/%s\n", $file;
       print $url, "\n";
       system("curl -O $url");
       $error=$_;
       print "$error\n";
       system("rm $file") if $error =~ /No such/ or $error =~ /File not found/ or -s $file < 16;
    }
 } 

Linux Resource Troubleshooting Commands

Command that reveal the inner workings of our machine

ulimit
free
vmstat
netstat
sysstat
mpstat
iostat
ping -c 1
traceroute 
sar

What is the process consuming the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2


What is the memory usage after 16:30 ...
   sar -s 16:30:00 -r
What is the disk paging after 16:30 ..
   sar -s 16:30:00 -B
Another day on I/O and mem usage
sar -f /var/log/sa/sa12 -u -n SOCK -B -r -s 15:30:00 -e 16:10:00

If you used previous versions of jvmstat you may be familiar with some of the older names... Here is a mapping of the old names to the new names:
JDK 5.0
jvmps is now jps, sse docs
jvmstat is now jstat
perfagent is now jstatd
jvmstat 3.0
visualgc is still visualgc

Detecting Running a VMWare Clone Linux

Detecting if your OS is running on a vmware machine

( grep -i vmware /proc/scsi/scsi || /sbin/lspci | grep -i vmware ) && echo This is a Virtual Machine

Eclipse over XWindows

Running and rendering the remote XWin session of eclipse.

61  set | grep SSH
[me@remote ~]$ set | grep SSH
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
SSH_CLIENT='172.18.153.150 49238 22'
SSH_CONNECTION='172.18.153.150 49238 10.10.10.3 22'
SSH_TTY=/dev/pts/3

Use the SSH_CONNECTION ip listed on local to export

60  xhost +  
70  ssh -X foouser@ip_remote_host 'export DISPLAY=172.18.153.150:0;/opt/eclipse-3.3.1.1-europa/eclipse/eclipse'

Do not use the hostname or worse alias for the remote but use the remote ip address in our example 10.10.10.3 When you do not have a local Xserver running you might see this:... if thats the case change your myhostnameasalias to the network IP address of the remote host, 10.10.10.3..

me@local $ ssh -X 'me@myhostnameasalias' 'export DISPLAY=172.18.153.150:0;java -Xmx1536m -Xms1024m -jar myjar.jar '
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Warning: No xauth data; using fake authentication data for X11 forwarding.
Jun 21, 2012 12:01:18 PM java.util.prefs.FileSystemPreferences$2 run
INFO: Created user preferences directory.
Exception occurred in main() of MyProgram
java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:169)
 at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)
 at sun.awt.X11.XToolkit.(XToolkit.java:89)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:169)
 at java.awt.Toolkit$2.run(Toolkit.java:834)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:826)
 at sun.swing.SwingUtilities2$AATextInfo.getAATextInfo(SwingUtilities2.java:120)
 at javax.swing.plaf.metal.MetalLookAndFeel.initComponentDefaults(MetalLookAndFeel.java:1556)
 at javax.swing.plaf.basic.BasicLookAndFeel.getDefaults(BasicLookAndFeel.java:130)
 at javax.swing.plaf.metal.MetalLookAndFeel.getDefaults(MetalLookAndFeel.java:1591)
 at javax.swing.UIManager.setLookAndFeel(UIManager.java:514)
 at javax.swing.UIManager.setLookAndFeel(UIManager.java:554)

To solve this ....