Python Search Command Output

This is the place for queries that don't fit in any of the other categories.

Python Search Command Output

Postby qwaven » Thu Jun 26, 2014 7:08 pm

Hey all,

I'm pretty new to Python, however I've been trying to write a script that first connects to server via SSH, then issues and command to read the contents of a file. <-- this works

My issue now is that I really don't want all of the contents of the file. I'd like to pass the output from the SSH command and then print out only lines matching particular search terms. I'm having a lot of trouble getting this to work and I'm sure this can't be that complicated. :)

All the examples I've found online pretty much involve a filename.txt to start with. However in my case I'm trying to read a file through SSH and then only display what I want. Hoping someone can provide me some sort of example on how I can achieve this. :)

I've included my code below with only the reading of the file part since nothing I was doing past that would work. :)

Code: Select all
#!/usr/bin/python

import paramiko
import cmd
import re
import sys

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('HOSTIP', username='myusername', password='mypassword')
stdin, stdout, stderr = ssh.exec_command("cat /path/to/my/text/mytext.txt")
stdin.flush()
data = stdout.read()

print data


Hope someone can help!

Cheers
Last edited by micseydel on Thu Jun 26, 2014 7:12 pm, edited 1 time in total.
Reason: First post lock.
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby micseydel » Thu Jun 26, 2014 7:14 pm

So the code you've posted here works without errors, and the only problem is that you want to do some manipulation of the string contained in your data variable?
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1256
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Python Search Command Output

Postby qwaven » Fri Jun 27, 2014 2:21 pm

Hey Micseydel,

Thanks for the reply. Yes that is correct, except its not a single string it would be a the output from cat on a textfile. For example I am hoping to print specific entries within 'data' instead of all of data. Say any line that contains "this is a good line" or "hello". :)

Thanks
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby Yoriz » Fri Jun 27, 2014 3:20 pm

It might be easier to help , if you could give an example of what the text input would contain and then an example of what output from that you would like to see.
for example would the text allready be in seperate lines or would it need splitting first, does the line need to start with a certain text or just contain some text somewhere in the line ect ect.
New Users, Read This
Join the #python-forum IRC channel on irc.freenode.net!
Spam topic disapproval technician
Windows7, Python 2.7.4., WxPython 2.9.5.0., some Python 3.3
User avatar
Yoriz
 
Posts: 782
Joined: Fri Feb 08, 2013 1:35 am
Location: UK

Re: Python Search Command Output

Postby qwaven » Fri Jun 27, 2014 7:07 pm

Hi Yoriz,

The text would yeah mostly be on separate lines. The lines may have data all spaced by tabs or similar however.

Basically I'm trying to do similar to egrep command where you can grep based on a keyword and output the whole line.

So any line that contains "this text like this" will be spit out.

Example:

This world is a round world
The dog walked about the park with this text like this as it was going home 12345
12345
12345

I'd only see:

The dog walked about the park with this text like this as it was going home 12345

Any help would be great. I think mostly I'm trying to determine what syntax to use/formatting...etc.
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby micseydel » Fri Jun 27, 2014 11:00 pm

Why not just use (e)grep instead of cat over SSH? Alternatively, you can use a list comprehension or filter() on data.splitlines().
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1256
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Python Search Command Output

Postby qwaven » Thu Jul 03, 2014 6:34 am

Thanks for the reply. Sorry for my late response. :)

When I try and use egrep I do not seem to get any results. Perhaps my syntax is wrong? When I issue the same command directly on the box it works just fine.

Code: Select all
stdin, stdout, stderr = ssh.exec_command("egrep my first line of text to find which has spaces|another one /path/to/my/file.txt")


Thoughts? Also not familiar with how to do the other suggestions. (very new to Python)

Thanks!
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby qwaven » Tue Jul 08, 2014 2:52 pm

no one?
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby qwaven » Wed Jul 09, 2014 5:24 pm

Ok realized I was missing some quotes. :)

I've got my ssh command working now. Something like below...

Code: Select all

stdin, stdout, stderr = ssh.exec_command('egrep "my first line of text to find which has spaces|another one" /path/to/my/file.txt')



However now I am having trouble with printing specific information.

For example my original command will bring back a bunch of information like:

computer name 1 xyzbn1234
uptime is 12345
this info: ghjjk
that info: 1234
computer name 2 mnbvytr567
uptime is 6665
this inf: jghyt
that info: 7657
computer name 3... etc.

What I'd like to do is filter this so I can print only the lines from Computer Name 1 to above Computer Name 2 and so on. Any ideas?

Thanks!
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby micseydel » Wed Jul 09, 2014 8:47 pm

qwaven wrote:What I'd like to do is filter this so I can print only the lines from Computer Name 1 to above Computer Name 2 and so on. Any ideas?

Please give an exact sample string and an exact example of what you want to do with it. I don't know what you mean by "only the lines from Computer Name 1 to above Computer Name 2 and so on".
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1256
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Python Search Command Output

Postby qwaven » Thu Jul 10, 2014 3:32 pm

Hi micseydel,

Thanks for the reply. Sorry for the confusion.

Let me try again. :)

Using my example from before. Lets assume that my ssh command returns the contents of a txt file located on a server. Currently I can print the contents okay.

Code: Select all

...
stdin, stdout, stderr = ssh.exec_command("cat /path/to/my/text/mytext.txt")
stdin.flush()
data = stdout.read()

print data



Lets say the contents of mytext.txt is filled with various computer inventory information.

Code: Select all

computer name 1 xyzbn1234
uptime is 12345
this info: ghjjk
that info: 1234
computer name 2 mnbvytr567
uptime is 6665
this inf: jghyt
that info: 7657
computer name 3... etc.



What I am asking/trying to do now is take the output from mytext.txt and only print certain parts of it. For example let's say I only want to print Computer 2's info. "cut" it out of the original data via some sort of pattern matching or based on lines in the file. Lines 4,5,6,7 would be Computer 2 in this example (assuming line starts at 0).

So:

Code: Select all

computer name 2 mnbvytr567
uptime is 6665
this inf: jghyt
that info: 7657



First off I'm not really sure on how to approach this. Like what kind of function should I be using? And if there is an example I can look at?

Hope that helps.

Thanks
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby qwaven » Mon Jul 28, 2014 6:52 pm

Hey all,

Not sure if anyone is still following this post, however I've made some progress.

I've skipped trying to get an pattern matching as I couldn't get it to work at all. I'm now using readlines()[#:#) and print. Where #:# are the line numbers in the text file I want to print.

So basically my code is something like this:

Code: Select all

import statements
(ssh in used in another script to download the text file locally)

then I open the file:

myfile = open("myfilename.txt", "r")

myfileparse = myfile.readlines() [55:75]

print myfileparse



So now that this is pretty much working I'm built many scripts like the above to grab the specific info I'm looking for. Granted the lines in the text file never change. Why I wanted to use pattern matching before. :)

Anyway my issue now is that since I am using readlines() my output is riddled with line break characters...etc which makes my output look pretty messy.

Sample: showing hosts that are connected: The \n and u' should not be there and does not appear to have any line breaks when opened in Windows...etc.
Code: Select all

[u'hostname1\n', u'Local Peers:\n', u' Origin-Host: abc.hostname2.com, Connected: true\n', u'Remote Peers:\n', u' Origin-Host: abc.hostname3.com, Connected: true\n', u' Origin-Host: nnn.hostname4.com, Connected: true\n', u' Origin-Host: nnn.hostname5.com, Connected: true\n', u'


Since I'm actually taking this output and opening via Apache webserver (html) I'd really prefer to just convert the \n into <br> or something so that all lines appear on new lines. I''ve seen some examples hear and there but nothing seems to have worked for me yet. So if anyone has any thoughts/examples...etc for how I can cleanup my output and/or put in proper line breaks that would be fantastic!!

Cheers!
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby snippsat » Mon Jul 28, 2014 7:02 pm

I'd really prefer to just convert the \n into <br> or something so that all lines appear on new lines.

Code: Select all
>>> lst = [u'hostname1\n', u'Local Peers:\n']
>>> [i.replace('\n', '<br />') for i in lst]
[u'hostname1<br />', u'Local Peers:<br />']
User avatar
snippsat
 
Posts: 163
Joined: Thu Feb 21, 2013 12:04 am

Re: Python Search Command Output

Postby qwaven » Mon Jul 28, 2014 9:18 pm

Hey Snippsat,

Thanks for the info. I guess I'm a little confused. With your example you are creating a list first. However with my script my output is from a variable.

Code: Select all

print myfileparse



So when I try and do what you wrote except with my variable it does not appear to work and I just get the same text with \n...etc

Code: Select all

lst = data

[i.replace('\n', '<br />') for i in lst]



So I'm guessing there is something I need to do instead in order to read from the variable? I also noticed there are no spaces between the line and the \n if that matters? Connected: true\n',

Hope you can help!

Thanks
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby snippsat » Mon Jul 28, 2014 9:48 pm

However with my script my output is from a variable.

A variable can hold any type(string,list,dict....)
print type(myfileparse) is it a list?
It look like a list in your post.
User avatar
snippsat
 
Posts: 163
Joined: Thu Feb 21, 2013 12:04 am

Re: Python Search Command Output

Postby qwaven » Wed Jul 30, 2014 6:14 pm

I see:

Code: Select all

<type 'list'>



My output seems to have a lot added to it from the readlines.

For example there is [' at the start and \n'] at the end. This seems to be around what would normally be each line. not sure if that helps much.

Any ideas?

Thanks!
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby snippsat » Wed Jul 30, 2014 9:50 pm

My solution's should work you are getting a list output,have you tried it?
My output seems to have a lot added to it from the readlines.

It's the normal output readlines() add \n.
Code: Select all
>>> lst = [u'hostname1\n', u'Local Peers:\n', u' Origin-Host: abc.hostname2.com, Connected: true\n']
>>> for item in lst:
...     print item
...     
hostname1

Local Peers:

 Origin-Host: abc.hostname2.com, Connected: true

>>> for item in lst:
...     print item.strip() #Remove \n as a test
...     
hostname1
Local Peers:
Origin-Host: abc.hostname2.com, Connected: true

>>> #Make a new list so "new line" work for html
>>> lst = [i.replace('\n', '<br />') for i in lst]
>>> lst
[u'hostname1<br />',
 u'Local Peers:<br />',
 u' Origin-Host: abc.hostname2.com, Connected: true<br />']

>>> for item in lst:
...     print item
...     
hostname1<br />
Local Peers:<br />
 Origin-Host: abc.hostname2.com, Connected: true<br />

Test in jsfiddle as you see new line now work in html.
User avatar
snippsat
 
Posts: 163
Joined: Thu Feb 21, 2013 12:04 am

Re: Python Search Command Output

Postby qwaven » Thu Jul 31, 2014 7:20 pm

Ok so I've tried again and this time I think its working. Not really sure what I did wrong before. (sorry about that)

I tried doing the same thing again to try and clean up the beginning of the lines and for some reason its not working for that part. Perhaps I don't quite understand the syntax were using here?

For example my lines right now start with either:
Code: Select all

[u'

or

', u'



I believe those were also added from the readlines.

Code: Select all

lst = [i.replace('\n', '<br />') for i in lst]
lst = [j.replace('', u'', '-->') for j in lst]



So what is wrong with my replace? Also is there a better way to remove the beginning part? I really don't need to replace it with anything, just remove.

Thanks!
qwaven
 
Posts: 11
Joined: Thu Jun 26, 2014 6:37 pm

Re: Python Search Command Output

Postby snippsat » Fri Aug 01, 2014 5:36 am

You don't need to replace u,just work with(u) as it was str.
You get back unicode(u) from server wish is not so strange.
With Paramiko 1.13.0, the method exec_command now returns Unicode. ...
Therefore, we must now ensure the encoding passed to the parser is utf-8.

unicode is text representation in characters.
str is text representation in bytes.

Code: Select all
>>> 'hello world'.decode('utf-8')  # str to unicode
u'hello world'
>>> 'hello world'.encode('utf-8')  # unicode to str
'hello world'

>>> s = 'hello world'
>>> u = u'hello world'
>>> type(s)
<type 'str'>
>>> type(u)
<type 'unicode'>
>>> s == u
True

For more reading look here
User avatar
snippsat
 
Posts: 163
Joined: Thu Feb 21, 2013 12:04 am


Return to General Coding Help

Who is online

Users browsing this forum: Google [Bot], snippsat and 4 guests