show me lines between two line markers / identifiers with perl

Start makers = 'begin - remote_properties' End markers = 'end - remote_properties'

perl -ne 'next unless /begin - remote_properties/ .. /end - remote_properties/;print' testfile.tibco_beGenerator

reading 3 lines in one time in oneliner perl

  cat PDDV_hosts.txt | perl -e 'while (my $frs=<>) { chomp $frs; $sec=<>; chomp $sec; $th=<> ;printf "%s,%s,%s", $frs, $sec, $th; }'

email all users that have exceeded their /home/ size quota with oneliner in du, perl

du -skh /home/* 2> /dev/null | sort -h | perl -pe 'if (/^\d{3}M/){ ($sz,$who) = $_=~ m!(\S+)\s\/home\/[a-z]{1,2}(75\d+)$!mxi;if ($who) {open(MAIL, "|/usr/sbin/sendmail -t");print MAIL "To: $who\@company.com\nFrom: mightymouse\@company.com\nSubject: homedir size exceeds: $sz; please remove items from your /home/ directory on uwb00078\n\nhomedir current size exceeds $sz > 99.9M; please remove items from your /home/ directory on uwb00078!";close(MAIL);print "$who $sz\n"}} $_=undef'

Perl find all modules on your INC path.

 perl -e 'foreach (@INC) {
    print `find $_ -type f -name "*.pm"`;
 }'

oneliner: list names of all ascii files with find and grep

find .  -type f -exec grep -lIRn RA1.1 {} \; | tee list

Remove arbitrary columns from text lines with perl and splice

Probably not the shorted ... remove the 9th and 0th column from an ls with perl and splice

find . -type f -ls | sort -k11 | perl -ne '@x=split;splice @x,9,1;splice @x,0,1; print join("\t", @x), "\n";' | tail 

List All Perl Modules That Are Installed On My Current System

One liner:

perl -MFile::Find=find -MFile::Spec::Functions -Tlwe 'find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC'

onliner - JAXBElement XJC Bindings and XMLRootElement creation for any schema

Some schema's do not get generated with the XMLRootElements which can get you into trouble marshalling Java 2 XML. See the error below

Caused by: javax.xml.bind.MarshalException
 - with linked exception:
[com.sun.istack.internal.SAXException2: unable to marshal type "com.tibco.xmlns.applicationmanagement.ApplicationType" as an element because it is missing an @XmlRootElement annotation]
 at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:317)
 at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:243)
 at org.springframework.oxm.jaxb.Jaxb2Marshaller.marshal(Jaxb2Marshaller.java:626)
 ... 25 more
Caused by: com.sun.istack.internal.SAXException2: unable to marshal type "com.tibco.xmlns.applicationmanagement.ApplicationType" as an element because it is missing an @XmlRootElement annotation
 at com.sun.xml.internal.bind.v2.runtime.XMLSerializer.reportError(XMLSerializer.java:216)
 at com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:286)
 at com.sun.xml.internal.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:462)
 at com.sun.xml.internal.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:314)
 ... 27 more
[ERROR] 

With the XJC bindings we can fix this however can be tedious going thru all the elements.... Boring.... so lets make a oneliner.... Here a one liner that will solve that and will generate the annotation section for your xcj.bindings or jaxb.bindings... Generating the XMLRootElement on bindings (requirements for perl will be XML::Simple see cpan)

perl -MXML::Simple -e 'BEGIN { use Data::Dumper; } $x=XMLin("./src/main/schema/5.5/ApplicationManagement.xsd");foreach (keys %{$x->{"complexType"}}) { printf "<jaxb:bindings node=\"xs:complexType[\@name=%c%s%c]\">
<annox:annotate><annox:annotate annox:class=\"javax.xml.bind.annotation.XmlRootElement\" name=\"%s\"/></annox:annotate></jaxb:bindings>\n",0x27,$_,0x27,$_ if $_; };'

Generates the output on the console:

<jaxb:bindings node="xs:complexType[=ActionType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="ActionType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=ServiceType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="ServiceType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=ServiceInstanceType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="ServiceInstanceType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=anyObject]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="anyObject"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=ApplicationType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="ApplicationType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=NVPairType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="NVPairType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=RepoInstanceType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="RepoInstanceType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=RemoteRepoInstanceType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="RemoteRepoInstanceType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=EventType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="EventType"/></annox:annotate></jaxb:bindings>
<jaxb:bindings node="xs:complexType[=BWServiceType]"><annox:annotate><annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement" name="BWServiceType"/></annox:annotate></jaxb:bindings>
Signings-MacBook-Pro:tibco-appmanage-schema $ 

THe end result as a XJC bindings file... which can be included in your XJC generation, in my case the maven plugin:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<jaxb:bindings xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
  xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:annox="http://annox.dev.java.net"
  xsi:schemaLocation="http://java.sun.com/xml/ns/jaxb http://java.sun.com/xml/ns/jaxb/bindingschema_2_0.xsd"
  version="2.1">
  <jaxb:bindings schemaLocation="ApplicationManagement.xsd"
    node="/xs:schema">
    <jaxb:globalBindings generateIsSetMethod="true"
      collectionType="java.util.ArrayList">
    </jaxb:globalBindings>
    <jaxb:bindings node="xs:complexType[@name=ActionType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="ActionType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=ServiceType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="ServiceType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=ServiceInstanceType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="ServiceInstanceType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=anyObject]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="anyObject" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=ApplicationType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="ApplicationType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=NVPairType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="NVPairType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=RepoInstanceType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="RepoInstanceType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=RemoteRepoInstanceType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="RemoteRepoInstanceType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=EventType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="EventType" />
      </annox:annotate>
    </jaxb:bindings>
    <jaxb:bindings node="xs:complexType[@name=BWServiceType]">
      <annox:annotate>
        <annox:annotate annox:class="javax.xml.bind.annotation.XmlRootElement"
          name="BWServiceType" />
      </annox:annotate>
    </jaxb:bindings>
  </jaxb:bindings>
</jaxb:bindings>

And the declaration of the maven plugin....

      <plugin>
        <groupId>org.jvnet.jaxb2.maven2</groupId>
        <artifactId>maven-jaxb2-plugin</artifactId>
        <version>0.8.3</version>
        <executions>
          <execution>
            <id>generate-domain1</id>
            <goals>
              <goal>generate</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <strict>true</strict>
          <schemaDirectory>src/main/schema/${schema.version}</schemaDirectory>
          <schemaIncludes>
            <value>scm-tibco-project.xsd</value>
          </schemaIncludes>
          <bindingIncludes>
            <include>xjc.bindings</include>
          </bindingIncludes>
          <verbose>true</verbose>
          <extension>true</extension>
          <args>
            <arg>-Xannotate</arg>
            <arg>-Xequals</arg>
            <arg>-XtoString</arg>
            <arg>-XhashCode</arg>
          </args>
          <generateDirectory>${project.build.directory}/generated-sources/xjc
          </generateDirectory>
          <plugins>
            <plugin>
              <groupId>org.jvnet.jaxb2_commons</groupId>
              <artifactId>jaxb2-basics</artifactId>
              <version>0.6.0</version>
            </plugin>
            <plugin>
              <groupId>org.jvnet.jaxb2_commons</groupId>
              <artifactId>jaxb2-basics-annotate</artifactId>
              <version>0.6.0</version>
            </plugin>
          </plugins>
        </configuration>
      </plugin>

using find and report on several lines with grep like functionality

Oneliner for finding a specific release of an artifact in a maven build file and reporting the artifact name and version even though these pieces are in different lines... And using find with the exit to print out only the successful files.

$ find . -name pom.xml -type f -exec perl -ne 'BEGIN {my $l} if (/2.0.1.RELEASE/) { printf "%s%s",$last,$_ ; $l=$last } ; $last=$_; END { exit 1 unless  $l; }' {} \; -print
   fmon-jms
   2.0.1.RELEASE
./master-service/master-service/master-proc/master-proc-core/pom.xml

find all the dependencies

usr@host:~/BUILD/foo-bar-root
$ find . -name 'pom.xml' -print -exec perl -ne 'BEGIN {my @k;} if (// ... /<\/plugin>/) { push @k, $_; } else { @k=undef;};  if (@k and grep {/<\/plugin>/} @k and grep {/pmd/} @k ) {print join("",@k);@k=undef}  ' {} \; -print 

Zip up my logfiles between 10 and 15 days old from a remote host...

Sledgehammering the logs archiving for perticular days and using zip... Tar always gives me a headache because of the -c vs -u option... why can't it just create the tar... anyway with zip its smoother and "smaller"!!

$ cat list  | xargs -i% -t ssh % 'find ./logs/ -mtime +10 -mtime -15 -exec zip -@ /tmp/my.zip "{}" \; -print '

Where list contains

me@host1
me@host2
me@host3

Capture Contents from Match until the end of the file - in perl

I want the classpath that is set in the manifest.mf

Class-Path:  mscorlib.jar Microsoft.VisualBasic.jar rdVbScriptTranslat
 or.jar js.jar rdsuperscript.jar System.Xml.jar rt.jar System.jar Syst
 em.Data.jar System.Web.jar rt.jar J2SE.Helpers.jar rt.jar fop.jar

The whole file is

$ cat META-INF/MANIFEST.MF 
Manifest-Version: 1.0
VMW-Assembly: c%3A%5CReportDev%5CVB%5CrdXslFoTemplate%5Cbin%5CDebug_Ja
 va%5CrdXslFoTemplate.dll
Created-By: MainSoft Inc
Version-Date: Mon 02/20/06 15:49:54.479
Assembly-Version: 1.0.0.0
Build-Version: 2.2.0.130
Class-Path:  mscorlib.jar Microsoft.VisualBasic.jar rdVbScriptTranslat
 or.jar js.jar rdsuperscript.jar System.Xml.jar rt.jar System.jar Syst
 em.Data.jar System.Web.jar rt.jar J2SE.Helpers.jar rt.jar fop.jar

$ 

What to do.... make the 3 dot match and make the last match \cD which is code for end of file....

$ find . -name 'rdXsl*.jar' -type f | while read dir; do [ -f META-INF/MANIFEST.MF ] && rm -f META-INF/MANIFEST.MF;unzip -q $dir META-INF/MANIFEST.MF; cat META-INF/MANIFEST.MF | perl -ne 'print if /^Class-Path/ ... /\cD/' ; done;

Where Did My Office Computer Go? Scan IP ranges for a service

So I lost my computer, since the network MAC->IP cache was flushed, however I know the subnet and the machine is on. But its a headless piece, no monitor. Where oh, where did my computer go??!?!?! Wireshark is more for recording what comes in, other network utils are on a IP basis... but lets curl ourselves out of here... I want to validate the existence of a twiki site:

curl -f -m 2 http://10.10.10.[100-200]/twiki/bin/view

Yeah its that easy.... and easy to forget...

grab that oneliner for sledgehammering with xargs and remote execute

Finally found the command on one node.... now how to grab that and execute it on milions of hosts (with a passwordless setup)

   66  ps auxww| grep Server | grep -v grep | awk '{ print $2 }' | xargs -i{} -t /usr/sbin/lsof -p {}  | grep wls10

Here you go

   77  history | grep Server | grep -v history | cut -c8- | grep wls10 | head -1 > ~/wls10.txt
   78  cat list | xargs -i{} -t ssh {} "$(<wls10.txt)"

xargs and using the stdin reference more than once

Multiple tar extractions in different directories based on the tar name.

computer:x usr$ ls -rtl
total 426552
-rw-r--r--  1 usr  usr  73451520 Jun 12 17:59 l01-2.tar
-rw-r--r--  1 usr  usr  72622080 Jun 12 17:59 l02-2.tar
-rw-r--r--  1 usr  usr  72284160 Jun 12 17:59 l03-2.tar

The only way I was able to do it is to push it into a small script, louzy.... I'll keep you posted on the alternatives.

computer:x usr$ cat test.txt 
mkdir $1 
cd $1 && tar xfv ../$1.tar && cd ..
computer:x usr$ ls -1 *.tar | sed 's/\.tar//' | xargs -I{} -t /bin/bash test.txt {} 

remote shell qoutes and double qoutes in complex operations

I prefer the sledgehammer when managing multiple servers. but what about quoting and double quoting and double double qouting.... escaping in on thing.... however how about this.... Here a oneliner for printing the statistics on TCP state of a box.

[dswown@pghpviswl01 ~]$ cat cmd.txt 
netstat | perl -MData::Dumper -ne 'BEGIN {%h=();} chomp;$_=~s/\s+/ /g;@x=reverse split/[\s+:]/ if /tcp|udp/; if (@x) {$x[2]=~s/\.c?\w?\w?$//g;print Dumper(\@x), $_ if $x[2] eq '8471';$h{count}{$x[0]}++;$h{$x[0]}{$x[2]}++;
} END { print Dumper(\%h)}'

[dsw@pviswl01 ~]$ cat list | xargs -i{} -t ssh {} $(<cmd.txt)

Where the file 'list' contains the following:

user1@hostA
user2@hostB

Virtual Host or Virtual IP dig digging

cmb@pscmbldl02:~
$ dig axfr my.domain.com | grep  dje00197.my.domain.com
dje00197.my.domain.com. 21600 IN A 10.76.40.34
djeswk03.my.domain.com. 21600 IN CNAME dje00197.my.domain.com.

cmb@pscmbldl02:~
$ dig axfr my.domain.com | grep pscmbldl02
scm.my.domain.com. 21600 IN CNAME pscmbldl02.my.domain.com.
scmblog.my.domain.com. 21600 IN CNAME pscmbldl02.my.domain.com.
pscmbldl02.my.domain.com. 21600 IN A 10.75.82.3
cmb@pscmbldl02:~
$ 

sed: how do I grep from line X until EOF?

If I have a rolling logfile with multiple lines per log record how do I print from line X until EOF?

computer:~ me$ sed -n "57450,\$p" pf/app/logs/server/trace.log

SED: print only lines between x and y...

Print only lines between certain line numbers... I'm sick of looking this up every two seconds... so I plagiarize and put it in my own collection.... which I know how to find :-)

# print section of file based on line numbers (lines 8-12, inclusive) 
sed -n '8,12p' # method 1 
sed '8,12!d' # method 2

Example that I needed this for: grepping lines 39000 - 40000 where an error was reported.

computer:tmp user$ ssh user@host 'cd  /QIBM/UserData/WebSphere/AppServer/V61/ND/profiles/dmgrapp/logs/dmgr && sed -n '39000,40000p' systemout.log'

A poor man's DNS resolving in a perl oneliner

Silly to record this ... but I hate spending more than 60 seconds on figuring out something I already figured out before... so here we go.... my own dns resolver..... Listing the resolving of the ip range: 10.75.19.80 until 10.75.19.95

$ perl -e 'for (80..95) { open EXEC, "( nslookup 10.75.19.$_ )|"  or die $!; @l=;chomp @l; ($x) = grep {/name/} @l if @l; printf "%3d - %s\n", $_, $x }'