Strip or honor ASCII color coding

[jessehouwing]

Agent Version and Platform

2.133.3
Windows | Linux

VSTS Type and Version

VisualStudio.com

If VisualStudio.com, what is your account name? http://psd30.visualstudio.com

What's not working?

When a commandline tool issues ASCII colour coding information (e.g. �[?25l) they are shown in the logs and make it harder to read.

2018-05-19T15:43:16.7926995Z It looks like this is your first time using Cypress: 2.1.0
2018-05-19T15:43:16.7938560Z 
2018-05-19T15:43:16.7958404Z �[?25l[15:43:05]  Verifying Cypress can run /opt/vsts/_work/r1/a/xxx-CI/src/node_modules/cypress/dist/Cypress [started]
2018-05-19T15:43:16.7980465Z [15:43:06]  Verifying Cypress can run /opt/vsts/_work/r1/a/xxx-CI/src/node_modules/cypress/dist/Cypress [completed]
2018-05-19T15:43:16.8000804Z �[?25h
2018-05-19T15:43:16.8019478Z Opening Cypress...

It would be useful to either honor the color coding, somehow instruct/configure the terminal as monochrome or stripping out the nonsense characters.

[bryanmacfarlane]

Yeah, this is on our backlog. labelling as an enhancement.

[adamralph]

Any update? As the maintainer of a build package which uses ANSI colours, I'm starting to get questions.

[TingluoHuang]

we are fixing this step by step.

1. get all UTF-8 characters back correctly Set code page to 65001 (UTF8) #1975
2. the Web UI might already support color coding. :)

[adamralph]

Thanks for the info. AFAIK the web UI still doesn't support ANSI colour codes, see https://dev.azure.com/blairconrad/SendComics/_build/results?buildId=9&view=logs&jobId=011e1ec8-6569-5e69-4f06-baf193d1351e&taskId=b8cbf579-6a08-5e02-2788-e25b5bd52b1b&lineStart=13&lineEnd=14&colStart=1&colEnd=1

@blairconrad you reported adamralph/bullseye#150 - are you still seeing this problem in the web UI?

[blairconrad]

@adamralph, I see no change. ANSI codes show up in the web UI:

https://dev.azure.com/blairconrad/SendComics/_build/results?buildId=13

[yaananth]

The web UI supports minimal set of escape codes, but the full support is coming soon.
We are prioritizing this.
Since this is UI issue and not agent issue, feel free to post something here
https://developercommunity.visualstudio.com/spaces/21/visual-studio-team-services.html?type=idea

But we understand the request and are working on it.

[yaananth]

Changes should be rolling out. We support ANSII color code now.
Here's what we support:

Graphics mode (m) -
0 (reset),
1 (bold),
3 (italic),
4 (underline),
22 (not bold),
23 (not italic),
24 (not underline),
39 (default fg),
49 (default bg),

fg colors - 30 (black), 31 (red), 32 (green), 33 (yellow), 34 (blue), 35 (magenta), 36 (cyan), 37 (white), 90 (grey)
bg colors - 40 (black), 41 (red), 42 (green), 43 (yellow), 44 (blue), 45 (magenta), 46 (cyan), 47 (white)

[TingluoHuang]

Close this issue base on this colorful screenshot. :D

[blairconrad]

PRETTY!

[blairconrad]

Not to be a Negative Nelly, but I notice that while the completed build output looks nice and colourful, the in-progress output does not deal well with the ANSI codes:

(from the in-progress build of https://dev.azure.com/blairconrad/blair%20bullseye/_build/results?buildId=18)

[yaananth]

That's a known limitation now unfortunately, we don't use the same view for both views :(
This is getting addressed though, just won't be super soon :)

[blairconrad]

Oh, thanks for the speedy response. Glad it's known!

[blairconrad]

I hate to complain, but as @adamralph noted over at adamralph/bullseye#199 (comment), in some cases the combination of the ANSI escape codes and < or > in the output confuse the renderer. I have created a minimal reproduction which you can see at https://dev.azure.com/blairconrad/azure-ansi/_build/results?buildId=22.

You can see the HTML produced for the ANSI-less and ANSI-ful output here:

<div style="top:192px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8739531Z&nbsp;No&nbsp;colour</span></span>
</div>
<div style="top:208px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8741003Z&nbsp;&lt;options&gt;</span></span>
</div>
<div style="top:224px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8758154Z&nbsp;</span></span>
</div>
<div style="top:240px;height:16px;" class="view-line">
  <span><span class="mtk1">2018-12-14T19:46:38.8759918Z&nbsp;Yellow&nbsp;foreground</span></span>
</div>
<div style="top:256px;height:16px;" class="view-line">
  <span><span class="mtk1"><span>2018-12-14T19:46:38.8761200Z </span>&zwnj;<span class="ansifg-y "><options></options></span>&zwnj;</span></span>
</div>

It seems that once the ASNI codes are used, <options> is translated into <options></options> instead of <options>.

Is https://developercommunity.visualstudio.com/spaces/21/visual-studio-team-services.html?type=idea still the right place to mention this or is there a better place?

[yaananth]

Here are the known limitations with this feature -

• Visually text won't appear selected if you try to select the text (but copy-paste works)
• Live feed won't honor color codes
• Color codes with < > tags will make them disappear

This is something we decided to live with until we address all of them together in the next spike which is planned soon. We decided to go ahead with the feature even with those as it's much better than showing some random useless characters :)

If you are curios on the limitations, we use monaco editor, which isn't really made for "viewing" experience. So to bring color coding support, we had to have those limitations for the interm...

Thanks for reporting and we understand the issues you mentioned and we plan to work on those, let us know if you find anything else.

For this particular issue, I think having it here makes sense as everything is being discussed here, but in general yes, dev community is the place to track these.

[blairconrad]

Thanks for the response @yaananth. I had doubts about https://developercommunity.visualstudio.com/spaces/21/visual-studio-team-services.html?type=idea as the "idea" at the end sounds more like feature requests, not bug reports.

I eagerly await the next release and appreciate all your efforts.

And I realize that every client is different, but given the choice between "bonus funny characters in the output" and "some sections of the output just omitted altogether", I'd've picked the second. The recovery mode of the first is "just try not to look at the funny characters" while for the second it's "hopefully realize something is wrong and then pore over the source HTML".

[yaananth]

You can report a problem: https://developercommunity.visualstudio.com/content/problem/post.html?space=21

[adamralph]

@yaananth earlier you said:

We decided to go ahead with the feature even with those...

So you have knowingly introduced a bug. Why do you need someone to "report" it?

[olafurpg]

Visually text won't appear selected if you try to select the text (but copy-paste works)

@yaananth Is there an open issue to fix this limitation? I searched in the Visual Studio Community but couldn't find a related discussion. Copy-paste works for in-progress builds but it stops working when the colors have rendered after the build completes. To reproduce, try copy-pasting line 4000 from the CmdLine entry in https://dev.azure.com/olafurpg/Metals/_build/results?buildId=51

[sbachstein]

Have there been any advances regarding the character encoding not being resolved?

[yaananth]

Multistage pipelines feature will bring new views which will honor ansii codes, evening when live streaming!

[lindluni]

While coloring is better with the multistage feature enabled, codes are still dumped in the output of the multistage build

[yaananth]

We support https://en.wikipedia.org/wiki/ANSI_escape_code#3/4_bit
It's possible the tool is using other specifications, eg: https://en.wikipedia.org/wiki/ANSI_escape_code#8-bit

Which we don't support yet.
However we should be parsing them out.
So, looks like there's some bug there.
Could you send a repo text file or link to where it happens (if it's a public run).
Thanks!

[smeathers]

Debugging a Teraform pipeline has lead me here. Like others I can make the logging more readable with the -no-color switch but the lack of colour in the output still reduces the readability of what I'm looking at. It would be a huge help if this could be supported.

[jwedel]

Would also appreciate this, as Cypress output in releases windows is pretty much unreadable:

[ransagy]

Another chime in hope someone from MS responds to this in any way, including "this is backlog/icebox/what".
@bryanmacfarlane @TingluoHuang @yaananth @vijayma @jtpetty - Can anyone point us at the right direction here?

[anatolybolshakov]

Hi everyone, this issue does not seem to be related to the agent itself (since agent does not handle logs view, but only sends them to server) - for the Azure DevOps features/issue reports I would suggest to open a ticket on https://developercommunity.visualstudio.com/search?space=21 to get right eyes on it.

[wwestrop]

For anyone following the thread: https://developercommunity.visualstudio.com/t/Colorful-logs-in-Release-Pipeline-as-wel/1652600

[ghost]

@jessehouwing why is this closed? I'm still seeing this issue, particularly on the "raw logs" view of Azure CI pipelines.

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale

[MattJeanes]

This is still an issue unfortunately

[FreddyAyala]

Hello, this is still a big issue, are we actively working on it? If you give me access to the internal Repo here in MS I can try to give a hand. Thanks.

[solidcloudio]

Energizer bunny... Still going..

2023-07-27T03:17:59.8405393Z Resource and property changes are indicated with these symbols: 2023-07-27T03:17:59.8405937Z �[38;5;208m-�[0m Delete 2023-07-27T03:17:59.8406280Z �[38;5;77m+�[0m Create 2023-07-27T03:17:59.8406611Z �[38;5;141m~�[0m Modify 2023-07-27T03:17:59.8407101Z �[38;5;246mx�[0m NoEffect 2023-07-27T03:17:59.8407695Z �[0m=�[0m NoChange 2023-07-27T03:17:59.8408184Z �[38;5;246m*�[0m Ignore

[williamdphillips]

Any progress on this?

[FreddyAyala]

Bump, still very important feature missing.

[martingalvan]

@FreddyAyala I understand you work for Microsoft, correct? Could you solve this internally, or talk to whatever team's responsible for this?

[mloskot]

Dear Azure team, any news on fixing this annoyance?

or, do you really expect people to keep working around it with, for example:

cat log.txt | sed -e $'s/\x1b\[[0-9;]*m//g' > log.clean.txt

[mloskot]

The feedback I've just received from Azure DevOps to my request posted at https://twitter.com/mloskot/status/1760636981028016191 seems to suggest that begging Azure for attention here is a waste of time:

We understand how you feel. However, at the moment, our dev community is the best place to report issues or suggest ideas to our product team.

[dewolfs]

Looking forward to get this fixed

[MaxSolabAzb]

One more voice to get this fixed. Or at least give me an option to opt out of the colours so that my logs don't look like dog's vomit. And adding $PSStyle.OutputRendering = "PlainText" does not seem to do anything.

[Depact]

Another one bumps the thread. Looking forward to be fixed

[Saulopv]

+1