Trailing space in heredoc delimiter

[p5pRT]

Migrated from rt.perl.org#128557 (status was 'open')

Searchable as RT128557$

[p5pRT]

From @epa

Created by @epa

The end marker for a here-document must be the exact string, without
trailing spaces. If the heredoc begins with <<END but instead of
'END' you have 'END ', Perl treats that as part of the quoted string
without warning.

I appreciate that this is done to be consistent with Unix tradition.
The Unix shell likewise requires an exact line with the terminator
string and no trailing space. However, I suggest that in this day and
age it may be worth a small break with tradition in order to catch the
mistake of a trailing space after the terminator.

Firstly, many commonly used text editors do not show any visual
difference between a line with trailing space and one without, so two
programs which look identical end up with different behaviour.

Secondly, we can no longer assume that Perl programmers are familiar
with the dark arts of Unix shell scripting, so they will not know
about this 'gotcha'.

Thirdly, Perl has already broken with the behaviour of the shell in
the case of a carriage-return character. A trailing carriage return
after the heredoc terminator is silently ignored, even on platforms
where text files use just a newline character for end of line. The
Unix shell does not do this (or at least GNU bash doesn't). Given
that we have already broken with the classical shell heredoc behaviour
for one kind of whitespace character, why not for others?

In my personal opinion the right fix would be to add a deprecation
warning when the heredoc terminator is seen followed by whitespace and
end-of-line. 'In future Perl versions this will be treated as the end
of the heredoc.' Then that change can indeed be made after a release
cycle or two.

An alternative would be to simply add a warning when the heredoc
terminator appears but has whitespace after it. Although I appreciate
the comments that heredocs are intended to be unrestricted with
'anything' appearing in them, I feel the case for a warning is even
stronger here given the invisibilty of the trailing space characters.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.22.2:

Configured by Red Hat, Inc. at Mon Jun 13 12:25:51 UTC 2016.

Summary of my perl5 (revision 5 version 22 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=4.4.9-300.fc23.x86_64, archname=x86_64-linux-thread-multi
    uname='linux buildvm-14.phx2.fedoraproject.org 4.4.9-300.fc23.x86_64 #1 smp wed may 4 23:56:27 utc 2016 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Dldflags=-Wl,-z,relro  -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro  -Dlddlflags=-shared -Wl,-z,relro  -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.22.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='  -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fwrapv -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='5.3.1 20160406 (Red Hat 5.3.1-6)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-Wl,-z,relro  -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib
    libs=-lpthread -lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lresolv -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.22'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro '
    cccdlflags='-fPIC', lddlflags='-shared -Wl,-z,relro  -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    Fedora Patch1: Removes date check, Fedora/RHEL specific
    Fedora Patch3: support for libdir64
    Fedora Patch4: use libresolv instead of libbind
    Fedora Patch5: USE_MM_LD_RUN_PATH
    Fedora Patch6: Skip hostname tests, due to builders not being network capable
    Fedora Patch7: Dont run one io test due to random builder failures
    Fedora Patch15: Define SONAME for libperl.so
    Fedora Patch16: Install libperl.so to -Dshrpdir value
    Fedora Patch22: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015)
    Fedora Patch26: Make *DBM_File desctructors thread-safe (RT#61912)
    Fedora Patch27: Make PadlistNAMES() lvalue again (CPAN RT#101063)
    Fedora Patch28: Make magic vtable writable as a work-around for Coro (CPAN RT#101063)
    Fedora Patch29: Fix duplicating PerlIO::encoding when spawning threads (RT#31923)
    Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux
    Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux


@INC for perl 5.22.2:
    /home/eda/lib64/perl5/
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5
    .


Environment for perl 5.22.2:
    HOME=/home/eda
    LANG=en_GB.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_GB.UTF-8
    LC_MESSAGES=en_GB.UTF-8
    LC_MONETARY=en_GB.UTF-8
    LC_NUMERIC=en_GB.UTF-8
    LC_TIME=en_GB.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/eda/bin:/home/eda/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin:/home/eda/.local/bin:/home/eda/bin:/sbin:/usr/sbin
    PERL5LIB=/home/eda/lib64/perl5/
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @jkeenan

On Wed, 06 Jul 2016 13​:50​:48 GMT, eda@​waniasset.com wrote​:

This is a bug report for perl from eda@​waniasset.com,
generated with the help of perlbug 1.40 running under perl 5.22.2.

-----------------------------------------------------------------
[Please describe your issue here]

The end marker for a here-document must be the exact string, without
trailing spaces. If the heredoc begins with <<END but instead of
'END' you have 'END ', Perl treats that as part of the quoted string
without warning.

I am unsure as to the specific case you are discussing.

In the attachment, the first heredoc is defined with '<<END;'. The second is defined with '<<END ;' In both cases there is no whitespace after the terminator 'END'. Both "work" in the sense that both compile and print -- even though the second definition looks weird.

Is that the case you are concerned about? Or something else?

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

128557-heredoc-terminator.pl

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @Abigail

On Fri, Sep 29, 2017 at 07​:27​:58PM -0700, James E Keenan via RT wrote​:

On Wed, 06 Jul 2016 13​:50​:48 GMT, eda@​waniasset.com wrote​:

This is a bug report for perl from eda@​waniasset.com,
generated with the help of perlbug 1.40 running under perl 5.22.2.

-----------------------------------------------------------------
[Please describe your issue here]

The end marker for a here-document must be the exact string, without
trailing spaces. If the heredoc begins with <<END but instead of
'END' you have 'END ', Perl treats that as part of the quoted string
without warning.

I am unsure as to the specific case you are discussing.

In the attachment, the first heredoc is defined with '<<END;'. The second is defined with '<<END ;' In both cases there is no whitespace after the terminator 'END'. Both "work" in the sense that both compile and print -- even though the second definition looks weird.

Is that the case you are concerned about? Or something else?

I think the perceived problem is with a trailing space after the
terminator *after* the here doc. As in​:

  print << END;
  This is printed

  END
  This as well, because the previous line
  has a trailing space.

  END

I'm not sure how big a problem this is in practise. In many cases, a
trailing space will lead to a program which doesn't compile. For the
remaining cases, where both the program compiles and doesn't behave
badly, a linter (Perl​::Critic?) ought to do the trick.

Abigail

[p5pRT]

From @jkeenan

On Sat, 30 Sep 2017 16​:18​:58 GMT, abigail@​abigail.be wrote​:

On Fri, Sep 29, 2017 at 07​:27​:58PM -0700, James E Keenan via RT wrote​:

On Wed, 06 Jul 2016 13​:50​:48 GMT, eda@​waniasset.com wrote​:

This is a bug report for perl from eda@​waniasset.com,
generated with the help of perlbug 1.40 running under perl 5.22.2.

-----------------------------------------------------------------
[Please describe your issue here]

The end marker for a here-document must be the exact string,
without
trailing spaces. If the heredoc begins with <<END but instead of
'END' you have 'END ', Perl treats that as part of the quoted
string
without warning.

I am unsure as to the specific case you are discussing.

In the attachment, the first heredoc is defined with '<<END;'. The
second is defined with '<<END ;' In both cases there is no
whitespace after the terminator 'END'. Both "work" in the sense that
both compile and print -- even though the second definition looks
weird.

Is that the case you are concerned about? Or something else?

I think the perceived problem is with a trailing space after the
terminator *after* the here doc. As in​:

print << END;
This is printed

END
This as well, because the previous line
has a trailing space.

END

I'm not sure how big a problem this is in practise. In many cases, a
trailing space will lead to a program which doesn't compile. For the
remaining cases, where both the program compiles and doesn't behave
badly, a linter (Perl​::Critic?) ought to do the trick.

I agree. I think the feature request should be rejected.

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @xsawyerx

On 09/30/2017 07​:27 PM, James E Keenan via RT wrote​:

On Sat, 30 Sep 2017 16​:18​:58 GMT, abigail@​abigail.be wrote​:

On Fri, Sep 29, 2017 at 07​:27​:58PM -0700, James E Keenan via RT wrote​:

On Wed, 06 Jul 2016 13​:50​:48 GMT, eda@​waniasset.com wrote​:

This is a bug report for perl from eda@​waniasset.com,
generated with the help of perlbug 1.40 running under perl 5.22.2.

-----------------------------------------------------------------
[Please describe your issue here]

The end marker for a here-document must be the exact string,
without
trailing spaces. If the heredoc begins with <<END but instead of
'END' you have 'END ', Perl treats that as part of the quoted
string
without warning.

I am unsure as to the specific case you are discussing.

In the attachment, the first heredoc is defined with '<<END;'. The
second is defined with '<<END ;' In both cases there is no
whitespace after the terminator 'END'. Both "work" in the sense that
both compile and print -- even though the second definition looks
weird.

Is that the case you are concerned about? Or something else?

I think the perceived problem is with a trailing space after the
terminator *after* the here doc. As in​:

print << END;
This is printed

END
This as well, because the previous line
has a trailing space.

END

I'm not sure how big a problem this is in practise. In many cases, a
trailing space will lead to a program which doesn't compile. For the
remaining cases, where both the program compiles and doesn't behave
badly, a linter (Perl​::Critic?) ought to do the trick.

I agree. I think the feature request should be rejected.

Agreed.

[p5pRT]

From @epa

Thanks for your responses. This bug report is based on real-world code; a lot of code (repeated SQL calls, or printing chunks of output) do have repeated heredoc uses, and if those all use the same delimiter (such as END) it is quite easy to silently include a trailing space -- not shown in most editors -- and end up with code that compiles but is wrong.

I filed the bug report because it's quite awkward that a space character (moreover, an end-of-line one, which are the most difficult to see) has such a large effect. But forget the suggestion of adding a warning. That gets bogged down in the usual back and forth about perlcritic (which I love, and run over the codebase regularly -- but not in every single edit-test cycle). Could I ask instead whether the whitespace-sensitivity here is really needed?

What would happen if heredoc delimiters became like every other language token and weren't affected by space characters after them? They will always be special because they do depend on start of line, but couldn't they be more relaxed about whitespace at the end? Nothing else in the language is as fussy, and we know that with the mixture of different editors in use, trailing spaces tend to creep into code over time. Wouldn't it be a sensible incremental improvement to allow whitespace after these heredoc terminators?

Note that the current behaviour is not exactly consistent​: if you put the start of the heredoc as '<<END ' with a trailing space, it doesn't get terminated by the 'END ' you specified. The trailing space on the starting heredoc delimiter is silently stripped. Why not strip on the closing delimiter too?

[p5pRT]

From @Abigail

On Sun, Oct 01, 2017 at 12​:10​:57PM -0700, Ed Avis via RT wrote​:

Thanks for your responses. This bug report is based on real-world code; a lot of code (repeated SQL calls, or printing chunks of output) do have repeated heredoc uses, and if those all use the same delimiter (such as END) it is quite easy to silently include a trailing space -- not shown in most editors -- and end up with code that compiles but is wrong.

I filed the bug report because it's quite awkward that a space character (moreover, an end-of-line one, which are the most difficult to see) has such a large effect. But forget the suggestion of adding a warning. That gets bogged down in the usual back and forth about perlcritic (which I love, and run over the codebase regularly -- but not in every single edit-test cycle). Could I ask instead whether the whitespace-sensitivity here is really needed?

What would happen if heredoc delimiters became like every other language token and weren't affected by space characters after them? They will always be special because they do depend on start of line, but couldn't they be more relaxed about whitespace at the end? Nothing else in the language is as fussy, and we know that with the mixture of different editors in use, trailing spaces tend to creep into code over time. Wouldn't it be a sensible incremental improvement to allow whitespace after these heredoc terminators?

Note that the current behaviour is not exactly consistent​: if you put the start of the heredoc as '<<END ' with a trailing space, it doesn't get terminated by the 'END ' you specified. The trailing space on the starting heredoc delimiter is silently stripped. Why not strip on the closing delimiter too?

This does what you expect it to do​:

  print << "END ";
  The line below does not end with a space​:
  END
  The line below does end with a space​:
  END

Abigail