Normal buffer issues

[nick.ace]

I'm trying to make a cel shaded shader and am working on the outline part. Everything seems to work well except for this:

 

http://steamcommunity.com/sharedfiles/filedetails/?id=603858840

 

It seems to me like the normals past a certain depth are not uniform and I'm not really sure why. Or is this just Nyquist-like effect? If so, what would be the best way to handle this?

 

This is the shader I wrote (there's a simple lua script that binds the diffuse, depth and normal textures):

 

#version 400
uniform sampler2D texture1; //diffuse
uniform sampler2DMS texture2; //depth
uniform sampler2DMS texture3; //normal
uniform vec2 camerarange;
uniform bool isbackbuffer;
uniform vec2 buffersize;
out vec4 fragData0;
#define width 2
float DepthToZPosition(in float depth) {
return camerarange.x / (camerarange.y - depth * (camerarange.y - camerarange.x)) * camerarange.y;
}
void main(void)
{
vec2 texcoord = gl_FragCoord.xy/buffersize + 0.5/(buffersize*0.5);
if (isbackbuffer) texcoord.y = 1.0 - texcoord.y;
vec4 c = texture(texture1, texcoord);

//Line Detection
bool edge = false;
float depth = DepthToZPosition(texelFetch(texture2,ivec2(texcoord*buffersize),0).x);
float depth_temp = 0;
vec3 normal = texelFetch(texture3,ivec2(texcoord*buffersize),0).xyz;
normal=normal/length(normal);
vec3 normal_temp = vec3(0);

// Check adjacent pixels
for (int x=-1; x<2; x+=2)
for (int y=-1; y<2; y+=2) {
depth_temp = DepthToZPosition(texelFetch(texture2,ivec2(texcoord*buffersize)+ivec2(x*width,y*width),0).x);
normal_temp = texelFetch(texture3,ivec2(texcoord*buffersize)+ivec2(x*width,y*width),0).xyz;
normal_temp = normal_temp/length(normal_temp);

if ((abs(dot(normal_temp,normal)) < 1) && (abs(depth_temp-depth) > .1)) {
edge = true;
}

}

fragData0 = c;

if (edge) {
fragData0 = vec4(0,0,0,c.a);
}

}

[nick.ace]

Nvm, I figured it out. The normals start to deviate by more than what I expected, so I added adapting cut offs.

[Josh]

You are normalizing the value you read, right? The stored value will not be normalized, in order to fit more resolution into a 32-bit image.

[nick.ace]

Yeah I'm normalizing them, but after a certain depth, the normals seem to change a little. Maybe it's due to floating point precision? Either way, I was able to find a workaround by comparing the dot product of adjacent normals with a threshold that changes based on depth.

[shadmar]

normals should be sampled like this I believe:

vec3 normal = normalize(texelFetch(texture3,ivec2(texcoord*buffersize),0).xyz * 2.0f - 1.0f);

 

also Leadwerks assign depth,diffuse,normals to texture0,1,2 default, so you don't really need a lua for the assigning if you use them in this order

[nick.ace]

I didn't know they were assigned by default. Thanks Shadmar!

 

Why is the sample coordinate being changed from (0 to buffersize) to (-1 to 2*buffersize-1)? Wouldn't it just be 0 to buffersize for the texel coordinate?

[shadmar]

It's just converting from [0,1] space to [-1,1] and normalized. It's not doing anything with the texture coordinates.

[nick.ace]

Oops I read the parentheses wrong

 

What's the point of changing the space if you don't mind me asking?

 

Just to clarify, I normalized the normal without the normalize command because I forgot that was a build-in command.

[shadmar]

This explains normal mapping pretty well and the diffrent spaces.

http://learnopengl.com/#!Advanced-Lighting/Normal-Mapping

[nick.ace]

Oh I see. Thanks for the link!