## I am having a problem to remove duplicates in a list.

This is the place for queries that don't fit in any of the other categories.

### I am having a problem to remove duplicates in a list.

The problem was given as remove duplicate numbers in a list. You must create a new list and you can't use remove. I have attempted use the following:

Code: Select all
`def remove_duplicates(numbers):    new_list = [] # To initialize new_list    numbers.sort()## To test if the list is empty#    if len(numbers) == 0:        new_list = []        return new_list## To test if there is only one integer in the list.#    if len(numbers) == 1:        new_list.append(numbers[0])        return new_list## To test if one integer is equal to the next integer# The range has to be one short or you will go over the index#    new_list.append(numbers[0])    j = 0    k = 1    while k < len(numbers)-1:        if numbers[j] != numbers[k]:            new_list.append(numbers[k])            j = j + 1            k = k + 1        if numbers[j] == numbers[k]:            j = j + 1            k = k + 1    return new_listnumbers = input("Enter a list of integers: ")print(remove_duplicates(numbers))`

It works for a few things. The problem I am having is that I keep getting index out of bounds if I change any the main section, for example while j < len(numbers)-1 where I substitute k with j or add an equal sign instead of the less than sign. If I have five elements in the list it will not add the fifth element for example [1,2,3,4,5] will give [1,2,3,4]. Like I said before, if I change it to get one more loop it gives me index out of bounds. I am at a loss. Surely, it would work with the right combination? It is the only problem I am having before I can move on to classes. Thanks, Micah.
Last edited by micseydel on Wed Apr 23, 2014 2:38 am, edited 1 time in total.
Reason: Locked OP.
ilikesci

Posts: 3
Joined: Wed Apr 23, 2014 1:38 am
Location: Fort Hood, Texas

### Re: I am having a problem to remove duplicates in a list.

Welcome to the forum! Thank you for using code tags by the way :)

I'm wondering what limits there are on this assignment of yours. Are you limited in terms of what Python features you can use? It looks like you're making a copy, but I wonder if the order needs to be maintained, if you're limited in terms of space or time for your algorithm, or if efficiency isn't a goal at all. You're doing an in-place modification as well, I assume this is allowed? (Just want to cover all the bases.)

With regards to your out of bounds error, you should provide the input you're giving to get it as well as the full traceback verbatim. That would be the whole output of your program, which would include the line number where that error occurs as well as the actual line.

Some other advice: indent your comments to the same level, since people used to reading Python will at first glance think you're in a new function at that point. It's just kinda icky. Also just as a lucky shortcut you can simplify your first two cases of len=0 or 1 to
Code: Select all
`if len(numbers) < 2: return numbers[:]`
Due to the reasons discussed here we will be moving to python-forum.io on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.

micseydel

Posts: 3000
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

### Re: I am having a problem to remove duplicates in a list.

You are massively over thinking the problem.
Don't sort the list.
Just itterate through it and add values you haven't seen yet to a new list. Of course, if you know your list will only ever contain hashable items (numbers, strings, tuples) it is even easier.

For starters:
Code: Select all
`new_list = []for item in old_list:    #Magic here`

-Mek
New Users, Read This
• Use code tags when posting code.
• Include any errors with your post (in code tags).
• Describe your problem; not your chosen solution.
• Make examples the minimum length to demonstrate your issue.

Mekire

Posts: 1711
Joined: Thu Feb 07, 2013 11:33 pm
Location: Tucson, Arizona

### Re: I am having a problem to remove duplicates in a list.

Thank you. Probably am over thinking it. I am testing it with [], [1], [4,5,5,4], and [6,8,122,2,18]. The first two if statements take care of [] and [1].
Code: Select all
`if len(numbers) < 2: return numbers[:]`

Does a better job then my six statements. Thank you.
I used the sort to get [4,5,5,4] to [4,4,5,5]. I think I was getting [4,5,4] as output one way I was trying it.
As it stands now the [6,8,122,2,18] will give me [2,6,8,18] leaving the 122 off which gave me the idea that it is with odd number of elements. It works for [1,1,2,2], it gives me [1,2]. It just has to include the 122. There is no limit on how long it takes or how long it is. Optimization is later. Having a grasp on basic concepts is the purpose for now, I assume. My pseudocode would read something like: the main idea was to compare two integers and see if they are equal. If equal, do not include the second one in the list. Then go to the next one. That is why j and k start off with different counts. Probably do not need the sort in the end. Input will be a list of integers. No floats, strings, or tuples in this one. This is the only problem that I have not been able to figure out.
ilikesci

Posts: 3
Joined: Wed Apr 23, 2014 1:38 am
Location: Fort Hood, Texas

### Re: I am having a problem to remove duplicates in a list.

Use a for loop. Start over. The approach you are taking is not what you want to do.
Itterate through the list. If the item is not in the new_list, add it. That's it.
The function body should be no more than five lines (and If we can assume only numbers, the body of the function is one line).

-Mek
New Users, Read This
• Use code tags when posting code.
• Include any errors with your post (in code tags).
• Describe your problem; not your chosen solution.
• Make examples the minimum length to demonstrate your issue.

Mekire

Posts: 1711
Joined: Thu Feb 07, 2013 11:33 pm
Location: Tucson, Arizona

### Re: I am having a problem to remove duplicates in a list.

I was given a suggestion and it works perfectly:

Code: Select all
`    while k < len(numbers):        if numbers[j] != numbers[k]:            new_list.append(numbers[k])            j = j + 1            k = k + 1        else:            if numbers[j] == numbers[k]:                j = j + 1                k = k + 1`

I am not quite sure why this works and the other way didn't. I am going to have to think about this. Your adding one more iteration but it doesn't seem like the else should matter and no index out of bounds.
ilikesci

Posts: 3
Joined: Wed Apr 23, 2014 1:38 am
Location: Fort Hood, Texas

### Re: I am having a problem to remove duplicates in a list.

Code: Select all
`    while k < len(numbers):        if numbers[j] != numbers[k]:            new_list.append(numbers[k])            j = j + 1            k = k + 1        else:            if numbers[j] == numbers[k]:                j = j + 1                k = k + 1`

Is equal to:
Code: Select all
`    while k < len(numbers):        if numbers[j] != numbers[k]:            new_list.append(numbers[k])        j = j + 1        k = k + 1`

Be very careful with phrases like "it works perfectly". Most likely in this case, it doesn't. Just because your code works for a few cases doesn't mean it has no bugs. You need to read and analyze your code so you understand it. Don't assume it works because it seems to.

The wording of the question is confusing. It looks like you're supposed to make a new list. In that case Mekire is absolutely right and a simple loop though the original list with a single test is sufficient. Why aren't you just doing that??? Are you familiar with the 'in' test?

Code: Select all
`l1 = [1,2,3]if 2 in l1:  # True if the number 2 is IN the list called l1    print "It's in there!"`

So, all you have to do is create an empty results list and then loop through the original list testing whether or not each item is already in your results list. If not, append it. Done. One loop, no sorting.

BTW, an even better solution is to use the SET container since it allows no duplicates but I assume that's not allowed yet (Shame on the teachers that hobble the language! There are better ways to teach the basics.):

Code: Select all
`def no_dups(in_list):    return list(set(in_list))`

Marbelous

Posts: 401
Joined: Fri May 31, 2013 8:12 pm

### Re: I am having a problem to remove duplicates in a list.

I almost forgot to mention the underlying reason your current attempt at solving this is confusing.

You are modifying the sequence you're iterating over WHILE you're iterating. You have a loop condition based on the length of a list called numbers but you are changing the size of that list in the loop itself. Do you know how python handles this? Even most advanced programmers avoid this problem (usually by iterating over a copy) because it's tricky and often introduces very subtle bugs.

Marbelous

Posts: 401
Joined: Fri May 31, 2013 8:12 pm

### Re: I am having a problem to remove duplicates in a list.

Marbelous wrote:BTW, an even better solution is to use the SET container since it allows no duplicates but I assume that's not allowed yet (Shame on the teachers that hobble the language! There are better ways to teach the basics.):

At my university, the way to inhibit you from using something like a set for this kind of problem was to have us use C. I'd rather they let them use a modern language and give some kind of restriction. The idea isn't to learn Python, the idea is to learn problem solving, which is certainly more important in the short term. There does come a point when it's silly to limit them, but in an intro course, although it's frustrating for advanced students it makes sense for the larger group.
Due to the reasons discussed here we will be moving to python-forum.io on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.

micseydel

Posts: 3000
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

### Re: I am having a problem to remove duplicates in a list.

micseydel wrote:I'd rather they let them use a modern language and give some kind of restriction. The idea isn't to learn Python, the idea is to learn problem solving, which is certainly more important in the short term.

I understand this philosophy, but I tend to disagree with it in this context. It enforces the idea that Python is merely a toy starter language that is used only because it is easy. If you want to teach C, use C.

If you want to use Python, I feel that you can teach core concepts perfectly well with out perverting or doing things that are unnatural to the language. To enforce the set restriction for example, rather than simply tell students what they can and can't use, create a logical requirement. Here all you need say is that the final result must maintain the same relative order as the original sequence; a perfectly reasonable request with such a problem.

-Mek
New Users, Read This
• Use code tags when posting code.
• Include any errors with your post (in code tags).
• Describe your problem; not your chosen solution.
• Make examples the minimum length to demonstrate your issue.

Mekire

Posts: 1711
Joined: Thu Feb 07, 2013 11:33 pm
Location: Tucson, Arizona

### Re: I am having a problem to remove duplicates in a list.

I disagree. My first C++ class prohibited the use of the STL. Using C just means there's nothing to prohibit. I'd rather they use a neutered Python for six or so weeks than have to use C as their first language and then decide they hate programming. (Although I did have an assignment to write add() and multiply() functions with bit operations rather than arithmetic operators, so I guess you can even neuter C.)

There are times when the prohibitions are too stringent, but this really isn't a supreme injustice. It's also hypothetical. Perhaps they just didn't each them sets yet, and he's not actually prohibited from doing anything. If there was a requirement to maintain the order, I'd still use a set, it just wouldn't be a one-liner. And they could still prohibit sets for that.
Due to the reasons discussed here we will be moving to python-forum.io on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.

micseydel

Posts: 3000
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

### Re: I am having a problem to remove duplicates in a list.

I don't think it's too much to ask that a teacher should come up with a way to ensure that her students learn the fundamentals of python's structures (i.e.: indexing and slicing) at the same time they allow and even encourage students to employ an object oriented approach. Both ways are just different tools for accomplishing the same task. Understanding all the options is a big picture approach but assuming it's taught correctly and students understand WHY to choose one way over another it gives them a sense of the full power of the language which should be a positive thing. Of course I'm not a teacher but there's my \$0.02.

Marbelous

Posts: 401
Joined: Fri May 31, 2013 8:12 pm

Return to General Coding Help

### Who is online

Users browsing this forum: Bing [Bot] and 6 guests