Description:
5.1.49:
ha_innodb_plugin.dll!read_view_sees_trx_id()[read0read.ic:70]
ha_innodb_plugin.dll!lock_clust_rec_cons_read_sees()[lock0lock.c:524]
ha_innodb_plugin.dll!row_search_for_mysql()[row0sel.c:4171]
ha_innodb_plugin.dll!ha_innodb::general_fetch()[ha_innodb.cc:5653]
mysqld.exe!rr_sequential()[records.cc:385]
mysqld.exe!sub_select()[sql_select.cc:11399]
mysqld.exe!do_select()[sql_select.cc:11150]
mysqld.exe!JOIN::exec()[sql_select.cc:1915]
mysqld.exe!mysql_select()[sql_select.cc:2519]
mysqld.exe!mysql_derived_filling()[sql_derived.cc:294]
mysqld.exe!mysql_handle_derived()[sql_derived.cc:56]
mysqld.exe!open_and_lock_tables_derived()[sql_base.cc:5057]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5058]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2288]
mysqld.exe!mysql_parse()[sql_parse.cc:6021]
mysqld.exe!dispatch_command()[sql_parse.cc:1256]
mysqld.exe!do_command()[sql_parse.cc:882]
mysqld.exe!handle_one_connection()[sql_connect.cc:1136]
mysqld.exe!pthread_start()[my_winthread.c:85]
mysqld.exe!_callthreadstart()[thread.c:295]
mysqld.exe!_threadstart()[thread.c:275]
kernel32.dll!BaseThreadStart()

interesting thing was that this was a segfault, not an assertion.

How to repeat:
only seen once, will give more details if it occurs again.

This bug looks serious. Affects 5.0, 5.1, and 5.5, plugin + builtin.
To repeat:

First create the schema:

drop table if exists t1;
create table t1(a int)engine=innodb;
insert into t1 values (1),(2);

Next, in two or more threads run the following 3 SQL continuously in a loop:

start transaction;
select * from t1 lock in share mode;
select * from `t1` group by polygon((select a from `t1` limit 1 for update) ,t1.a);

Verified just as described on Mac OS X, also with current mysql-next-mt. This is a stack trace:

macbook-pro:next-mr openxs$ tail -100 var/macbook-pro.err 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100727 20:38:27 mysqld_safe mysqld restarted
100727 20:38:27 [Warning] Setting lower_case_table_names=2 because file system for /Users/openxs/dbs/next-mr/var/ is case insensitive
100727 20:38:27 [Warning] One can only use the --user switch if running as root

100727 20:38:27 [Note] Plugin 'FEDERATED' is disabled.
100727 20:38:27 [Note] Plugin 'ndbcluster' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
100727 20:38:27  InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100727 20:38:27  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
100727 20:38:27  InnoDB: Warning: allocated tablespace 4, old maximum was 0
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
100727 20:38:27 InnoDB 1.2.0 started; log sequence number 1657360
100727 20:38:27 [Note] Event Scheduler: Loaded 0 events
100727 20:38:27 [Note] /Users/openxs/dbs/next-mr/libexec/mysqld: ready for connections.
Version: '5.6.99-m4-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
100727 20:38:27 - mysqld got signal 10 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=2
max_threads=151
thread_count=2
connection_count=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337863 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x1820000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0xb077ef34 thread_stack 0x30000
0   mysqld                              0x0054c93b my_print_stacktrace + 44
1   mysqld                              0x00104386 handle_segfault + 876
2   libSystem.B.dylib                   0x940472bb _sigtramp + 43
3   ???                                 0xffffffff 0x0 + 4294967295
4   mysqld                              0x003aaf96 _ZN4ListI19st_foreign_key_infoE9push_backEPS0_ + 54666
5   mysqld                              0x003f9645 _ZN4ListI19st_foreign_key_infoE9push_backEPS0_ + 375865
6   mysqld                              0x0039c064 init_heap_psi_keys + 350432
7   mysqld                              0x0039c1f2 init_heap_psi_keys + 350830
8   mysqld                              0x0021bb52 _Z13rr_sequentialP11READ_RECORD + 110
9   mysqld                              0x0018ca78 _Z10sub_selectP4JOINP13st_join_tableb + 316
10  mysqld                              0x0019c8f8 _ZN4JOIN9join_freeEv + 1566
11  mysqld                              0x001ac925 _ZN4JOIN4execEv + 2641
12  mysqld                              0x001a8f37 _Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex + 797
13  mysqld                              0x001ae545 _Z13handle_selectP3THDP3LEXP13select_resultm + 547
14  mysqld                              0x00115d13 _Z15update_precheckP3THDP10TABLE_LIST + 1075
15  mysqld                              0x001184f2 _Z21mysql_execute_commandP3THD + 2960
16  mysqld                              0x00120bfc _Z11mysql_parseP3THDPKcjP12Parser_state + 624
17  mysqld                              0x001217e0 _Z16dispatch_command19enum_server_commandP3THDPcj + 2616
18  mysqld                              0x00122c93 _Z10do_commandP3THD + 621
19  mysqld                              0x0010f053 _Z24do_handle_one_connectionP3THD + 425
20  mysqld                              0x0010f117 handle_one_connection + 37
21  libSystem.B.dylib                   0x9400c095 _pthread_start + 321
22  libSystem.B.dylib                   0x9400bf52 thread_start + 34
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x18a0610 = select * from t1 group by polygon((select a from t1 limit 1 for update) ,t1.a)
thd->thread_id=1
...

I've got a crash by running the following in 2 shell sessions (after creating table):

while true; do bin/mysql -uroot test -e 'start transaction; select * from t1 lock in share mode; select * from t1 group by polygon((select a from t1 limit 1 for update),t1.a);'; done

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/115378

3479 Georgi Kodinov	2010-08-10
      Bug #55580 : segfault in read_view_sees_trx_id
      
      The server was not checking for errors generated during
      the execution of Item::val_xxx() methods when copying
      data to the group, order, or distinct temp table's row.
      Fixed by extending the copy_funcs() to return an error
      code and by checking for that error code on the places
      copy_funcs() is called.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/115570

3479 Georgi Kodinov	2010-08-12
      Bug #55580 : segfault in read_view_sees_trx_id
      
      The server was not checking for errors generated during
      the execution of Item::val_xxx() methods when copying
      data to the group, order, or distinct temp table's row.
      Fixed by extending the copy_funcs() to return an error
      code and by checking for that error code on the places
      copy_funcs() is called. 
      Test case added.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/115643

3479 Georgi Kodinov	2010-08-13
      Bug #55580 : segfault in read_view_sees_trx_id
      
      The server was not checking for errors generated during
      the execution of Item::val_xxx() methods when copying
      data to the group, order, or distinct temp table's row.
      Fixed by extending the copy_funcs() to return an error
      code and by checking for that error code on the places
      copy_funcs() is called. 
      Test case added.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/115920

3483 Georgi Kodinov	2010-08-13
      Bug #55580 : segfault in read_view_sees_trx_id
      
      The server was not checking for errors generated during
      the execution of Item::val_xxx() methods when copying
      data to the group, order, or distinct temp table's row.
      Fixed by extending the copy_funcs() to return an error
      code and by checking for that error code on the places
      copy_funcs() is called. 
      Test case added.

Noted in 5.1.51, 5.5.6 changelogs.

The server was not checking for errors generated during the execution
of Item::val_xxx() methods when copying data to a group, order, or
distinct temp table's row. 

Setting report to Need Merge pending push to 5.6.x.

Bug #53627 was marked as a duplicate of this one.

Pushed into mysql-5.5 5.5.6-m3 (revid:alik@ibmvm-20100825102234-a3q8x0l7voa13ts3) (version source revid:alik@ibmvm-20100825102234-a3q8x0l7voa13ts3) (merge vers: 5.5.6-m3) (pib:20)

Already fixed in 5.5.x. Resetting to Need Merge.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/116850

3492 Sergey Vojtovich	2010-08-26
      Fixed race condition in a test case for BUG#55580.
     @ mysql-test/suite/innodb/t/innodb_mysql.test
        Fixed race condition in a test case for BUG#55580.

Pushed into mysql-trunk 5.6.1-m4 (revid:alik@sun.com-20100901130501-4g2k86dub29auj8y) (version source revid:alik@sun.com-20100901130012-9bmmvzcnnw6n5rw6) (merge vers: 5.6.1-m4) (pib:21)

Pushed into mysql-next-mr (revid:alik@sun.com-20100901130614-pgop3m80rmutewxn) (version source revid:alik@sun.com-20100901130033-8k19cjn6n2blm3py) (pib:21)

Pushed into mysql-5.5 5.5.7-m3 (revid:alik@sun.com-20100901125952-4hsrosoa0xreionr) (version source revid:alik@sun.com-20100901125952-4hsrosoa0xreionr) (merge vers: 5.5.7-m3) (pib:21)

Noted in 5.6.1 changelog.

Pushed into mysql-5.1 5.1.52 (revid:sunanda.menon@sun.com-20100928083322-wangbv97uobu7g66) (version source revid:sunanda.menon@sun.com-20100928083322-wangbv97uobu7g66) (merge vers: 5.1.52) (pib:21)

Pushed into mysql-5.1-telco-7.0 5.1.51-ndb-7.0.20 (revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (version source revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (merge vers: 5.1.51-ndb-7.0.20) (pib:21)

Pushed into mysql-5.1-telco-6.3 5.1.51-ndb-6.3.39 (revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (version source revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (merge vers: 5.1.51-ndb-6.3.39) (pib:21)

Pushed into mysql-5.1-telco-6.2 5.1.51-ndb-6.2.19 (revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (version source revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (merge vers: 5.1.51-ndb-6.2.19) (pib:21)

Already documented in the 5.1.51 changelog. Reverting to Closed state.